The Usability Testing industry has long stuck to the school of thought that comprehensive User Testing can still be achieved with as little as five users. Lets face it five isn’t much and when dealing with a demographic as widespread as the internet population it just doesn’t seem rational to suggest this is anywhere near enough. This has long been the school of thought though that even with just five Testing Users you could identify over 80% of your websites Usability issues.

We put this to the test with a Testing User study.

To begin with we selected 10 websites of past Testing User clients that varied both in size and nature of content. We selected participants from the Testing User panel in multiples of 5, 10 and 20 to establish what the ratio of improvement was (if any) in the Usability issues identified.

Firstly some background on where this idea that five participants is sufficient for User Testing came from. The first person to suggest this was Nielsen as a conclusion from one of his first studies into User Testing all the way back in 1993. Put simply he suggested that the statistical rigor could be considerably relaxed when conducting real world Usability Testing. By this he means that the Testing User participant numbers can be relaxed as he further went on to suggest in 2000 that elaborate lab controlled Usability Testing were a waste of resources.

This five participant theory has come under fire though. For example one study by Spool and Schroeder found that only 35% of Usability Issues were identified and furthermore they found that both from 13th participant and upwards at least one new Usability issue was discovered. A Usability study shortly after by Perfetti & Landesman discovered that when conducting a study involving 18 participants each new Testing User discovered at least five new Usability issues or obstacles.

So we got on the with the study. Now as the Testing User analysis is based on remote methodology, i.e: no laboratory conditions, we conducted the study this way. As is evident from the above 5 person User Testing has been dis proven as a comprehensive method over the years, so we thought it would be more beneficially to compare our ‘remote findings’ with that of previous laboratory tests. Now as the Testing User analysis differs from traditional usability methodology we decided that there did however need to be some sort of control in place to add validity to our findings. This required us to break away from the Testing User methodology and implement an ‘in person’ test of demographically selected users/participants that we could arrange to ‘come in’ and be observed working their way through the survey.

Our in house Usability Professionals had also identified all the Usability issues that were found after extensive analysis of the 10 websites earlier that month that yet to implement any design changes.

Now first of all the remote online method as you get if you purchased a Testing User Analysis. The data we received back showed that over the 10 websites tested, when compared to the issues identified the first 5 participants were able to record 29% of the usability issues our experts identified. The 10 and 20 Testing user groups were able to record 46% and 63% respectively.

In the in person observation the results were also inevitable. The User Testing survey questionnaire they had been set concluded that the first five participants recorded 32% of the Usability issues identified and the 10 and 20 Testing User groups were able to record 39% and 69% respectively. Now that’s obviously not completely conclusive as to what you would expect when comparing remote User Testing to controlled lab conditions, but it’s definitely not out of line neither.

We expect to publish the full report shortly but we can draw some preliminary findings.

Firstly Nielsen has suggested that the more users you add to a test, the less you will find and that larger Testing User participant numbers are only really necessary when the site has ‘highly distinct groups of users’. He may be right, but you don’t have to be a Usability Testing expert to know that this data is of course just individual opinions and that the more participants you use the more chance you have of a ‘clearer picture’ as to what your Usability issues actually are.

In his defense I think the purpose of the whole ‘five user theory’ is a pretty innocent attempt to just expand Usability Testing to a wider audience. I mean he critiques expensive lab tests quite strongly but he also suggests that you keep repeating these five person user tests at least three times.

We definitely agree with him that Usability Testing needs to be brought to a wider audience, if that was his intention, but we also think that the suggestion a comprehensive Usability Test can be conducted three times with five people is misguiding. There is only limited choice in the market though. Lets face it from the perspective of a small business you aren’t going to go to the trouble of setting up a controlled lab tests even if it was just 15 people. So you look at remote Usability Testing that’s more obtainable, say Video Testing. Still these cost an average of about $50 so with 15 participants that’s still $750 for some You Tube style videos of 15 users opinions where you have to extract and quantify the data yourself.

Lets not forget for the most part User Testing is still the luxury of large organisations with huge budgets for it. How are we supposed to sell Usability Testing and its benefits to a wider audience when the best the industry can come up with are these inadequate ‘do it yourself’ solutions?

The Testing User analysis was designed to confront the problem head on. Clients can now access the large numbers of participants previously only available for thousands of dollars. From as little as $125 clients can receive valuable insight into the Usability Issues that their web site faces.

The client does not have to make any sacrifices, once you have commissioned the test we do the rest.

We don’t stop there though.

Why should clients who want to conduct Usability& User Testing pay large amounts of money for data they have extract and quantify themselves? The Testing User analysis will arrive to you in a few days as a 30 page board room and presentation ready document outlining the strengths and weaknesses of Usability on your website. This is the direction Usability should be heading