After running design sprints for companies of all shapes and sizes in a wide range of industries over the last eighteen months or so, we’ve become familiar with some of the most common questions our clients have about design sprints and Differential’s process in general.

One of those questions is,

“Why do you only run five user interviews with the design sprint prototype?”

It’s a good question. Indeed, we only run five user interviews to test the design sprint prototype before determining which parts of the concept have been validated or invalidated.

At first, five interviews seems like not enough to get the kind of raw, actionable insights that the design sprint process purports to provide. This is especially true if you’re coming from an organization that relies heavily on big data in its decision making.

In order to address the small number of interviews, we need to keep in mind two things:‍

1. ‍User test recruiting - Part of the design sprint is dedicated to identifying the ideal target for the concept that is conveyed in the prototype so that we can run the user tests with the right types of users. If you’re stuck right now on the idea that your target user is “any person, between the age of 16 and 90 who eats bread,” or some other similarly broad persona, don’t worry. We’ll help you hone in on the best target for your concept and then we’ll deliberately recruit only those people who fit your target and screen people out if they don’t fit the target.
2. ‍The purpose - The interviews are great for getting quick, qualitative feedback on the concept so that we know how to proceed in developing that concept further.

This post, What’s the Point of the User Interviews, explores the purpose of these interviews further.

But why only FIVE?

The Math

The scholarship that supports five interviews as the magic number is strong.

‍As you test more and more, you learn less and less. ‍

Check out this graph produced by Jakob Nielsen that demonstrates that five test users typically uncover about 85% of usability problems based on binomial probability.

That being said, there are occasions, according to Neilsen, when you might want to run more than five user tests, for example if you have multiple, distinct groups of users or if you have the budget for extensive testing that must be spent. In these cases you’re better off running a series of small tests with iterations in between, rather than an elaborate everything-all-at-once kind of test.

Math not your thing? Think about it this way.

Let's say you're shopping for a new house, but you’re not quite sure what you want. So, you decide to go to a few open houses.

After the first open house, you've decided on the neighborhood. Seeing the second house convinces you that you really want a home office and the pristine condition of the third house helps you decide that the house you buy must be move-in ready.

You've only seen three houses and you've already honed in on some of the most important features - the location, size, and condition - that you'll look for as you continue your search.

If you go on to see another 20 or 30 houses, you're likely to get lost in the details of each one and forget what it was that excited you most about those first few houses. Congratulations. You’ve managed to create noise, the type that obscures the insights you gained early on.

Not only do we know, mathematically, that we get most of the valuable data early on, but we know this to be true from our experience, as well.

Here at Differential, it’s common that the fourth and fifth interviews just confirm what we heard in the first three and we rarely change our validated or invalidated determination as a result of the last two interviews.

For more info, see:

• Why 5 is the magic number for UX usability testing
• Why you only need to rest with five users (explained)
• Why test design with only 5 users?
• Determining Usability Test Sample Size