Paper Prototype Rocks

Last week we used a prototyping technique that has changed the way I look at development: paper prototyping.

I had heard a bit about this before and it sounded great… but there’s always such a drive to just start coding! Before I reach Carolyn Snyder’s excellent Paper Prototyping, I hadn’t realized what I was missing.

The key difference from what you might be thinking–and definitely what I was thinking–is that paper prototyping isn’t just about doing mockups of the UI and asking users for feedback. It means recreating the a subset of the core user experience using a paper-based model, with a human “computer” and the user interacting not with the facilitators or testers, but directly with the pieces of paper, by pointing for hovering, touching for clicking and writing for typing.

The results were amazing.

We took two weeks to develop & construct a paper prototype for our upcoming Search Map software, which is the first implementation of the SwitchBook approach to User-driven Search (which is the focus of our work with the VRM community). That was followed by one week doing the actual testing.

To build out a paper prototype like this you have to have a critically specific definition of what it is you are building. Depending on how early you are in the gestation of your software idea, this might be impossible. But if you have a pretty good idea of the basics, the paper prototype will force you to actually write down both

• a subset of tasks in sufficient detail for the user and the prototype to do with paper assets, and
• create the pieces of paper that represent every window, every menu, every dialog box, every state in the interface.

It felt like we went through three revisions of our core UI before we actually had any users. We found there were so many assumptions hidden in our initial requirements documentation that we were missing key parts of the user experience. And then, by mapping out those parts of the experience, we made significant leaps in simplifying and unifying the overall design.

Prior to the actual testing, we first did a dry run with a fellow associate as a user. This was eye opening. Not only did we identify a few assets we were missing, we realized we had several key assumptions that we had overlooked in our design. These assumptions led us to revise the paper prototype significantly, leading to an even better run through with the live testers.

This pattern repeated itself for everyuser. Snyder strongly suggested doing just a handful of tests, no more than two per day. So, 4-6 testers over two-three days. After two weeks building the prototype, I was thinking we should have more testers to really get the value out of this thing. But after the trial run, I realized that there is so much opportunity to update & improve between tests, that you want to immediately assimilate each session and update the prototype.

That is the real beauty of the paper prototype.  Because it is just paper… and in particular just sketches on paper–not a beautifully designed custom UI–it is trivial to change. We were evolving in leaps and bounds between every test, taking out portions of the interface, creating new icons, adding buttons here and removing them there, even updating dialog boxes in mid-test.

It would be hard to overstate how powerful it was to engage in such rapid evolution based on real user feedback. With ~3 revs in the construction phase, 1 with our trial run, and 4 more revisions from each test, I estimate a good 8 substantial revisions to our design in just three weeks.

Our timing for this exercise was just about perfect. We just started a 6 month dev cycle in January, after over a year of internal development (largely on the server side with a basic architectural prototype on the client). The core experiential basics had been fairly stable, and we were ready to integrate a lot of conceptual learning into a new rev. So, we had a lot of detail about what we wanted to do, and were in a good point to take a moment, document our requirements rigorously, and sit down with users to see what really works. For the record, we spent about three weeks in requirements engineering prior to working on the paper prototype (which was another three weeks of effort). I don’t know if I’ve ever spent a more useful 6 weeks in any programming project.

We also quickly saw the limitations of this approach. Of course, it was slow. We joked that the McKinney 5000 was operating at about 110hz. (Sam McKinney acted as our “human” computer for the testing–and did a great job I must say.) A good portion of our user experience depends on finding “flow” during advanced searches across many different websites and different search providers. Just keeping track of the user’s behavior was a breakneck task… simultaneously updating the screen to indicate the real-time feedback and recommendations from our server was even harder. The result is that although we got to test our core workspace, the technique was too slow to really test the work flow.

So, Carolyn Snyder, thanks so much for the easy and thorough guide to paper prototyping. It was an amazing exercise for our whole team. And for those of you who are working through the details with disruptively innovative software, I encourage you to try it. I think you too will be amazed at your results.

p.s.

My apologies that we didn’t get any pictures of the experience. We were so busy doing it, nobody stopped to capture the look & feel of the interactions.