User Testing vs Expert Evaluation

Expert evaluation looks at the complete system from many perspectives and reveals potential problems such as inconsistency, support for different ways of working, visibility of information and language use. It also enables elements such as error messages to be thoroughly investigated.

The benefits of expert evaluation

It is something of a given in the usability industry that testing with real users is the most effective way of identifying interface issues that may cause usability problems. Whilst it is true that testing is the best way to get real feedback on user experience with a site or application, there is an argument that evaluation by an expert is an equally efficient way to identify these problems.

For some reason, the process of expert evaluation in usability circles has acquired the name 'heuristic evaluation', a term that is largely inaccurate in this context and certainly confusing for those unfamiliar with the jargon. However, that is a different story. Here we are more concerned with the pros and cons of expert evaluation as compared with user testing.

The Case Against

Recent research has suggested that expert evaluation cannot be relied upon to identify 'real' interface problems. For example, Bailey (1992) reports that 46% of problems identified in an expert evaluation are real problems, 54% are false alarms, and about 20% of real problems are missed altogether. He concludes that "the only way to determine the real usability problems in a website is to conduct well-designed and professionally executed performance tests".

The Case For

However, there are a number of questions to be asked about this data and conclusion. The data says nothing about the seriousness of each problem. One usability problem may cause important users to be unable to complete a common task, whereas another may only cause a minority of less important users to take longer than necessary to complete a less important task.

This data also assumes that because a problem identified in an evaluation does not occur in a user test, it is therefore not a problem. This is an unusual way of looking at things - reminiscent of the driver who claims speeding is safe because "I've never had an accident".

Representing Minorities

Many problems will occur for a minority of users but not for any of those tested. Very rarely can a user test represent 95% of all users, though preventing 5% of your customers from buying is a serious issue for your business. In a heuristic evaluation, evaluators can look at the system from the perspective of the user population as a whole, anticipating problems that only a small minority would experience and only in particular circumstances.

Imagine if software engineers carried out functional testing by simply running through the major tasks in the kind of way most users would perform them. The result would be an extremely buggy product. User testing is nowhere near as comprehensive as software functional testing - expert evaluation can help to fill in the gaps.

Filling In The Gaps

A statement that 20% of problems identified in a user test are missed in an expert evaluation begs the question - how many problems identified in an expert evaluation are missed in a user test?

A user test will uncover the major problems that real users will have using the system to carry out the major tasks. However, there are likely to be a number of potential problems that are not uncovered by a user test, because:

None of the users in the test actually had the problem

None of the users in the test carried out the task in such a way that the problem arose

Casting an expert eye over the interface reveals some issues that may well cause problems for those user profiles perhaps not represented in the test participants. Even problems experienced by only 1% or 5% of users are worth resolving.

Similarly, there are usually many possible ways of carrying out any particular task. It may be that an interface supports some methods very well but for other methods it fails to provide the right information at the right time. A small number of users in a user test are not likely to cover all the ways that all users will carry out a task.

Conclusion

Expert evaluation looks at the complete system from many perspectives and reveals potential problems such as inconsistency, support for different ways of working, visibility of information and language use. It also enables elements such as error messages to be thoroughly investigated. In a user test, many potential error messages simply do not appear because no users make the error.