Is A Lab Essential For User Testing?

There should be no doubt that it is the testing itself that is most important to the success of a project. Although a usability lab is helpful, it is not essential. The lack of one should not deter an organisation from pursuing the user testing option.

Testing In The Lab

Once an organisation decides to go ahead with a user testing programme, the questions really begin. Is it really necessary to undertake testing in a 'usability lab'€ And what exactly should a fully functioning lab consist of anyway€ As one might imagine, opinion is divided on these issues. We take a quick look at what a typical lab might consist of and the pros and cons of lab-based testing.

What Is A Lab?

At the most fundamental level, a usability lab records a user tests and the events that occur within it for re-examination at a later date. In practise this tends to mean the collection of certain types of information including:

Video and audio of both the user and all on-screen activity throughout the test. User images are typically captured with a digital video camera, screen activity with a 'scan converter'. The two can then be passed through a video mixer for 'side-by-side' presentation. 'Clicks' and pathways through a site or application. A lab should be capable of recording a host of quantitative data concerning user activity on the site, both in terms of recording individual tests and combining multiple results as part of a statistical analysis. Key events and instances of defined user behaviours. These are typically defined before the test and will consist of any activity of particular concern or interest, such as 'user confusion' or 'mistake'. These activities can be tagged manually during a test, revealing how often, and where, they occur.

Of course, the true power of these separate data streams emerges when they are combined. Each instance of, for example, user confusion can be reviewed on video. An observable delay in user clickthrough can be analysed through observation of screen activity. Detailed reports illustrating typical usage patterns and times to task can be collated. A good usability lab should be at least capable of providing this sort of functionality. Put to good use, lab output can provide a number of benefits for testers:

The Ability To Review Information

By recording everything that occurs during a test, a lab report enables those conducting the test (or even those who didn't) to spend as long as is necessary analysing the results, without the risk of 'false memory' syndrome as time passes. The lab also enables deeper analysis of results and can assist in finding relationships between cause and effect that may otherwise go undetected. In short, this means a more accurate and thorough set of test results.

Improving the Test Procedure

The knowledge that all key events will be automatically recorded and can be observed later should help the tester concentrate on assisting the testee rather than frantically taking notes (a process which can also be distracting for both parties). This freedom can also help the tester focus on the general 'feel' of the test itself rather than any recording that would otherwise be required.

Providing Superior Test Reports

There is no doubt about it - video and audio evidence of user difficulties are more likely to convince development teams and managers of the need to re-examine interface design issues. Although conventional testing results may be just as accurate and worthwhile, the word of a usability expert is less likely to convince than the concrete evidence that is video output from a usability lab. For anyone selling usability within an organisation, lab results can be an invaluable aid.

Despite these obvious advantages to lab-based testings, some reasonable objections do exist:

Lab-based Testing Is Expensive

If you were building your own lab from scratch in order to conduct one round of tests, then the cost of a lab would certainly be prohibitive. However, given that simple lab facilities as described above can be hired from usability consultancies (including Frontend) much of this cost can be avoided. Alternatively, an organisation using a third party to conduct tests should find that use of a lab will not add significantly to quoted costs. After all, in many ways a lab will create efficiencies in the process that can even save money.

Lab-based Testing Is 'Unrealistic'

There is an argument that the fundamentally artificial nature of a lab-based user test will never be able to recreate real-world usage. However, in a sense this artificiality is inherent in any testing process, whether in a lab or not, and there is little evidence to suggest it has a major bearing on test results. Some labs are designed to minimise this effect, by placing testers in separate 'observation rooms' for example, although it is questionable how 'natural' an empty room flanked by a two-way mirror can be.

Summing Up

At Frontend we favour lab-based testing in most situations. Some of the arguments above demonstrate the benefits of the lab-based approach. But there should be no doubt that it is the testing itself that is most important to the success of a project. Although a usability lab is helpful, it is not essential. The lack of one should not deter an organisation from pursuing the user testing option.