Expert evaluation looks at the complete system from many perspectives and reveals potential problems such as inconsistency, support for different ways of working, visibility of information and language use. It also enables elements such as error messages to be thoroughly investigated.
It is something of a given in the usability industry that testing with real users is the most effective way of identifying interface issues that may cause usability problems. Whilst it is true that testing is the best way to get real feedback on user experience with a site or application, there is an argument that evaluation by an expert is an equally efficient way to identify these problems.
For some reason, the process of expert evaluation in usability circles has acquired the name 'heuristic evaluation', a term that is largely inaccurate in this context and certainly confusing for those unfamiliar with the jargon. However, that is a different story. Here we are more concerned with the pros and cons of expert evaluation as compared with user testing.
Recent research has suggested that expert evaluation cannot be relied upon to identify 'real' interface problems. For example, Bailey (1992) reports that 46% of problems identified in an expert evaluation are real problems, 54% are false alarms, and about 20% of real problems are missed altogether. He concludes that "the only way to determine the real usability problems in a website is to conduct well-designed and professionally executed performance tests".
However, there are a number of questions to be asked about this data and conclusion. The data says nothing about the seriousness of each problem. One usability problem may cause important users to be unable to complete a common task, whereas another may only cause a minority of less important users to take longer than necessary to complete a less important task.
This data also assumes that because a problem identified in an evaluation does not occur in a user test, it is therefore not a problem. This is an unusual way of looking at things - reminiscent of the driver who claims speeding is safe because "I've never had an accident".
Many problems will occur for a minority of users but not for any of those tested. Very rarely can a user test represent 95% of all users, though preventing 5% of your customers from buying is a serious issue for your business. In a heuristic evaluation, evaluators can look at the system from the perspective of the user population as a whole, anticipating problems that only a small minority would experience and only in particular circumstances.
Imagine if software engineers carried out functional testing by simply running through the major tasks in the kind of way most users would perform them. The result would be an extremely buggy product. User testing is nowhere near as comprehensive as software functional testing - expert evaluation can help to fill in the gaps.
A statement that 20% of problems identified in a user test are missed in an expert evaluation begs the question - how many problems identified in an expert evaluation are missed in a user test?
A user test will uncover the major problems that real users will have using the system to carry out the major tasks. However, there are likely to be a number of potential problems that are not uncovered by a user test, because:
None of the users in the test carried out the task in such a way that the problem arose
Casting an expert eye over the interface reveals some issues that may well cause problems for those user profiles perhaps not represented in the test participants. Even problems experienced by only 1% or 5% of users are worth resolving.
Similarly, there are usually many possible ways of carrying out any particular task. It may be that an interface supports some methods very well but for other methods it fails to provide the right information at the right time. A small number of users in a user test are not likely to cover all the ways that all users will carry out a task.
Expert evaluation looks at the complete system from many perspectives and reveals potential problems such as inconsistency, support for different ways of working, visibility of information and language use. It also enables elements such as error messages to be thoroughly investigated. In a user test, many potential error messages simply do not appear because no users make the error.
The Benefits of Viewing User TestsThe benefits of user testing have long been established. It is still important however to try and maximise these benefits. One way in which this can be done is by viewing the user test yourself.
Back To Basics: How Poor Usability Effects AccessibilityIn recent user testing with a range of participants including Visually Impaired (VIP) and Blind users we found that the majority of problems were common across all groups. However the effect of poor usability is more severe for users with visual disabilities. Surprisingly all of the issues are very familiar and are easy to fix so we thought we’d revisit some of the basics of accessible web design.
Webpage Layout: Right Hand Side BlindnessIn several recent websites we have user tested, the site designers have placed important task critical links and information on the right hand side (RHS) of three column page layouts. The user testing was conclusive, users ignore any information presented on the RHS. We think this is a similar effect to the well documented banner blindness. It is essential to ensure that import links or information is not positioned on the RHS as they will surely be ignored.
Introduction to User Centred Design ProcessUser Centred Design (UCD) is an approach to design where the end user is placed at the heart of the design and development process. It could even be described as a ‘Design Philosophy’. Knowing who your users are, what they want and if your system is fulfilling their needs is central to UCD.
Challenges For eGovernmentIn the same way that successful eCommerce companies have had to get to know their users - not just through focus groups but by involving them in design and development of products, services and interfaces - Government must get to know it's customers.
