Research Paper - Real or Imaginary: The effectiveness of using personas in product design

By Frank Long

This research paper was first published in the Irish Ergonomics Review, Proceedings of the IES Conference 2009, Dublin

ISSN 1649-2102  - Copyright 2009

Table of Contents

• Abstract
• Introduction
• Background
• Methadology
• Analysis
• Results
• Discussions and Conclusions
• References

Abstract

The use of personas as a method for communicating user requirements in collaborative design environments is well established. However, very little research has been conducted to quantify the benefits of using this technique. The aim of this study was to investigate the effectiveness of using personas. An experiment was conducted over a period of 5 weeks using students from NCAD. The results showed that, through using personas, designs with superior usability characteristics were produced. They also indicate that using personas provides a significant advantage during the research and conceptualisation stages of the design process (supporting previously unfounded claims). The study also investigated the effects of using different presentation methods to present personas and concluded that photographs worked better than illustrations, and that visual storyboards were more effective in presenting task scenarios than text only versions.

Introduction

The product development process for complex consumer products is a collaborative one. It involves multidisciplinary design teams, consisting of numerous individuals, working on different components of the design, often in geographically dispersed locations. It can be challenging to maintain a consistent user focus in such environments. User testing provides valuable design feedback, but generally this feedback arrives in the later stages of development when it is difficult to modify the design. In order to create more useable solutions, earlier user validation is required in the design process. The use of 'Personas' as a method for communicating user requirements and maintaining a consistent user focus in collaborative design efforts is becoming increasingly widespread. Personas are fictional user archetypes based on user research. Through a process of analysis and refinement, the data from user interviews is distilled into one or multiple fictitious characters. Each character is developed in realistic detail, and how that character wants to interact with the design is described as task scenarios. This Persona and their associated task scenarios form the basis for specifying how users want to experience the design. Through role-play and QA sessions using the persona, the merits of design solutions can be evaluated against the needs of the persona. Despite the growing popularity of personas, there is considerable debate as to the efficacy of the technique.

Background

There is a surprisingly little published work detailing the methodology or providing evidence of its effectiveness. At the time of writing only a few books deal with the subject in any detail. Cooper's 'The Inmates are Running the Asylum' (1999) provides a compelling business argument for using personas but offers little practical information on how to use them. 'About Face 2.0; The Essentials of Interaction Design' (Cooper and Reimann, 2003) details a more of a systematic overview of the process, strongly advocating their own trademarked design approach - Goal-Directed Design™. In 2006 Pruitt and Adlin published the most detailed volume on the subject, 'The Persona Lifecycle' – outlining and developing their interpretation of Cooper's original technique. This work focuses heavily on creation and maintenance of personas throughout the product lifecycle and includes case study references from industry. Pruitt's early background advocating and using personas in Microsoft is also documented through the short paper Personas Practice and Theory (Pruitt and Grudin, 2003). Mulder's 'The User is Always Right: a Practical Guide for Creating and Using Personas on the Web' (2007) focuses on the creation of personas and techniques for incorporating quantitative market research and other empirical data into the process.

From a research perspective, the problem with much of what has been published is the lack of objectivity. Most have adopted an advocacy standpoint and their examples and analysis tend to evangelise the use of personas. Furthermore there are distinct differences in the way the authors advocate using the technique. There is very little objective research or insight published. Chapman and Milham (also working at Microsoft) identify this in their 2003 paper:

"There is little peer-reviewed discussion of the personas method. The most comprehensive work to date (Pruitt and Adlin, 2006) is a compiled book advocating the method. There is no literature that systematically evaluates personas' methodology or utility in a comprehensive fashion."

"There have been no adequate studies addressing the reliability, validity, or utility of the method. To rectify this, we suggest the following as potential idea sketches for future research… (point 4)Assign multiple teams to design the same product, where some teams use personas and some don't. Which teams create products that are more usable?"
(Chapman and Milham, 2003, The persona's new clothes: methodological and practical arguments against a popular method, p2-4).

Methodology

The chief aim of the study was to assess and measure the effectiveness of using personas as a design tool - heeding Chapman and Milham’s call for further research. Does using personas as a tool give designers any advantage in designing more effective and user-centred solutions? Would there be a quantifiable usability difference between the designs from the persona-based teams over ones that did not use persona? A secondary element of the research was to investigate the effect of different presentation techniques on the effectiveness of the personas. Would the use of the storyboards with illustrated user personas over the traditional method of using realistic user photographs and text-based scenarios have any significant difference on the resulting designs?

At the core of this research is a project-based experiment that was conducted with a group of 3rd year Industrial Design students from the National College of Art and Design (NCAD) in Dublin. The experiment was planned as a design project conducted over a period of 5 weeks. The students were divided into 3 groups (see table 1.1). In order to ensure a 'level-playing field' the tutors were consulted to help distribute the class into groups of equal strength. Two groups used personas as a design tool and one group acted as the control group and did not use a persona-based approach. The objective of the experiment was to evaluate whether there was any difference in the usability of the final designs. The study also sought to measure how using personas influenced the design teams in their approach to solving the brief.

Table 1.1 Groups used in the experiment and associated design tools

Personas and Scenarios

The scope of the experiment was focused on the effectiveness of using personas in the design process. It was not practical for the students to research and create personas within the project timeframe. It was also felt that using different personas would add an additional, unwanted level of variability to the experiment. Instead, a pre-prepared user persona and task scenarios were given to the students (see fig 1.0 and 2.0). The persona matched the target-user demographic outlined in the project brief and care was taken to ensure that it did not contain any information that might create an unfair advantage over the control group. The persona detailed the personal information of a fictitious character, her likes and dislikes, her home environment and her attitude towards new technology. The persona documentation included a 'day in the life' section, where the character's daily routine and how she currently uses the product was described. Finally the persona's goals were listed, as were the client's business goals (these were also contained in the project brief). Two versions of the persona's image were used; the Beta group received a photograph of a real person while the Gamma group received an illustration. Two task scenarios 'Set-up' and 'Listening to the Radio' accompanied each persona - one set was text only the other was laid out as a storyboard. (See Fig 1.0 and 2.0)

The Project

Students were briefed in separate groups. Each group was instructed not to disclose or discuss their design tool to the other groups. The Beta and Gamma students were briefed in how to use personas as were their tutors. The tutors agreed to take observational notes during the project and a questionnaire was distributed at the end of the 5 weeks for the students and tutors to complete. Finally individual focus groups were conducted among the 3 design groups and the project tutors to discuss the experience of using the personas. The Beta and Gamma groups were asked to complete a recall test to capture how much detail that they could remember about each persona.

Fig 1.0 - Sample from the persona and task scenario documentation used by Beta team

Fig 2.0 - Sample from the persona and task scenario documentation used by the Gamma team

Analysis

On completion of the project, the grades awarded to the students at each stage and the overall grade was noted. By analysing the mean grades awarded to each group at the key project stages, it was possible to gain a comparative progress snapshot of the teams during the design process.

Detailed usability analysis was conducted on each individual design solution. The project timeframe did not facilitate the production of working prototypes or 3D models so user testing was not a viable method for evaluation. Instead a Heuristic evaluation was carried out on the designs using Nielsen’s usability heuristics (table1.2). By using a consistent checklist to evaluate each design a comparative score could be achieved. Understanding and measuring the variance between the projects was more important than gaining a definitive usability assessment of each one.

Table 1.2 - Nielsen’s Usability Heuristics

Each design was analysed against each heuristic, one at a time. As a design was first assessed for ‘user control and freedom’, if it contained a feature that supported this heuristic it scored 1 point, if it contained two or more features it scored 2 points. If no features supported that heuristic the score was 0. The analysis was carried out for each of the heuristics in this way. The overall score obtained by each design was tabulated and then the results were compared across the three groups.

Results

One of the key research questions was to determine whether using personas within the design process generates more user-friendly solutions. As usability is not a core criterion in the NCAD grading system, but only one of a large number of design attributes that are assessed, this question cannot be answered through simple comparative analysis of the students' final project results. Further analysis was carried out on each project using Nielsen's Usability Heuristics to isolate and quantify the usability attributes.

Table 1.3 - Heuristic evaluation scores by project team

The Heuristic analysis revealed a clear pattern. The teams that used the persona (Beta and Gamma) scored higher than the control group (Alpha). While the individual scores varied somewhat in accordance with the ability of the individual design teams, the overall results formed a consistent pattern. The Alpha group average was 2.67 compared to 5.67 (Beta) and 10.33(Gamma) as can be seen by table 1.3. These results indicate that the Beta and Gamma groups had a greater understanding of the user needs and designed solutions around the user-critical tasks to a greater degree than the control group.

Project Grades by Stage

Analysing the grades at each project stage showed that the gap between the mean scores for the control group and the groups using the personas was significantly larger (p value less than 0.05) at the research and concept stages. However, this gap had narrowed by the end of the project providing an even spread of overall grades.

From the results it is clear that the personas gave an advantage to the Beta and Gamma design teams during the early stages of the project. Through using personas these design teams had a clear perception of 'who' they were designing the solution for and what the users' goals were. Through role-playing scenarios and ideas they were able to quickly identify what design problems the user would encounter and to propose solutions that the user would want. As the project progressed the gap between the Alpha teams and Beta and Gamma teams narrowed. This suggests that by using their conventional design process the control group also identified and addressed many of the design problems; however it took them longer to reach this stage. The teams using personas seemed to identify and problem-solve design issues faster and earlier in the development process. The tutors confirmed this finding during the focus groups and feedback questionnaires.

Participant Feedback

The focus groups conducted at the end of the project confirmed that very little information regarding each design tool was leaked between teams. From the 16 responses to the questionnaire, 5 (31%) agreed that at the end of the project no information had leaked between groups, while 7 (44%) agreed that only very minor details had leaked out by the end of the project. The Alpha group (control) confirmed that they focused more on aesthetic aspects of the design brief and limited their consideration of user interaction to physical controls. The Beta group was generally enthusiastic about their experiences using personas. They incorporated the persona profile into their normal design approach – conducting additional research with real people that matched the profile of the persona, conducting product research in stores where the persona might shop etc. One key area where the Beta students identified the positive effects of using the persona was in relation to teamwork. It was considerably easier to achieve a design consensus within the team by being able to refer to an external target user. This allowed them to push personal opinions to the background.

"If one of us came up with a design feature that the other didn’t like we could think about it in terms of what Marie would like - it wasn’t just based on my opinion or their opinion, we could refer to the Marie; Marie would not like it to work like that" - Comment from Beta student during focus group

Using the persona seemed to increase the focus of the Beta team on the user interface issues. It was generally agreed that if they had not used the persona they would have arrived at a different design solution. When asked about using the approach again in the future all agreed that it would be useful. Some students admitted that if they had to make the persona themselves, they might be tempted to adjust the details to suit their own design outlook. The Gamma group were less enthusiastic (than the Beta group) about using the persona and through discussion it became clear that they did not empathise with 'Marie' to the same extent as the Beta group did. The group discussed the fact that their persona was represented by an illustration and therefore was obviously not a real person. This seemed to highlight the fact that the persona was based on research - and perhaps because the students were not involved in conducting the research, they were sceptical. The students also discussed how the illustrated image of the persona made them feel that they could pick and choose from her characteristics and even superimpose their own additional features and information onto her character. Some Gamma participants reported to ignoring parts of the persona and others stated that they designed their solution partly for the persona and partly for more general users. There was general agreement in the group that having the persona was helpful and that it helped prevent self-referential design.

Persona Memory Test

Both Beta and Gamma teams were given a memory test to identify whether there were any differences in their ability to recall specific details about the persona. While there was no statistically significant difference between the performances of the two groups, the Beta group (photograph) could remember more details (mean score 20.6) than the Gamma group (18.4). The difference was most apparent in relation to personal details such as age, names of family members, users likes and dislikes etc. Users with the sketch tended to give more vague answers - for example when asked her age some students responded with an age range 50-55 where as the beta group were more exact providing the precise age.

Tutor Feedback

As the tutors were aware of the research objectives and had a ringside seat during the project, they had a wealth of insights into how the student teams reacted to the various design tools. The overall impression from the tutors was that the persona-based approach definitely offered something different. The students seemed to be more enthusiastic about the design challenge when they could relate the product to a specific user. This also resulted in a higher quality and quantity of work produced by the students. The tutors agreed that the students found working and communicating within their teams easier through using the persona. Previously team-based projects had a tendency to result in conflict. While the tutors were very positive towards the technique, they expressed some reservations about using it in the future. Both expressed a concern that it the persona research was not done accurately then the design result could be badly compromised. Tutors felt that students could skew research to suit their own design agenda. They also noted that some students became so focused on the persona that they lost a little perspective on the general design issues. This would need to be balanced in future projects. The tutors reported that towards the end of the project, the general relevance of the personas within the teams began to wane. They suggested that as a result of this, the effect of the different tools diminished as the project progressed; the difference between the groups in terms of engagement with the user was considerable in the beginning and seemed to converge gradually during the course of the project. The tutors reported that at the outset of the project the good students of groups Alpha and Gamma seemed to under perform and weaker students in groups Beta and also, to some degree, Gamma performed better than normal. However during the course of the project both relative performances returned to their regular levels.

Discussions and Conclusions

The principle aim of this pilot study was to assess whether using personas as a design tool in the product development process would lead to more user-centred designs. It can be justifiably claimed at the outset that there are positive benefits associated with using the technique and this claim is evident from the analysis of the student design work, the student survey and the focus group feedback from the students and their tutors. This pilot study produced objective evidence to support the key claims made by Cooper, Pruitt et al. for using personas in the product design process. Personas strengthen the focus on the end user, their tasks, goals and motivation. Personas make the needs of the end-user more explicit and thereby can direct decision-making within design teams more towards those needs. Furthermore, the study suggests that using personas can improve communication between teams and facilitate more constructive and user-focused design discussion. Students using personas produced designs with better usability attributes than the students that did not use personas - thereby answering one of Chapman and Milham’s key concerns.

Using illustrations instead of photographs of the persona would seem to reduce effectiveness. It can lead to selective consideration of the persona characteristics and can increase the risk of self-referential details being superimposed onto the persona. The study also reported a lower level of empathy towards the illustrated persona and a diminished ability among students to recall details about the persona after time. The storyboard task scenario seemed to be more effective than the text version and facilitated more detailed design solutions.

The study indicates that using personas offers several benefits for user-centred design in product development. The results support claims that using personas focuses more attention on the end-user, particularly in the early stages of the project. Not only does this increase the likelihood of a more usable end product but it also provides a clear user focus at the initial design research and idea generation stages, which enhances the possibility of incorporating user-centred features at the product specification stage. Personas have a role in helping designers to innovate new ideas but can also assist in validating new designs as they emerge. They can provide a valuable user-centred input early in the development cycle - helping to bridge the gap between research and design.

Personas are only one of a number of user-centred design tools and are not suited to every situation. There are considerable resources required to implement a persona-based approach within a design project. The time and effort required to research, analyse and document personas, should not be underestimated. As can be seen from the early experiences at Microsoft, where timeframes and human resources are not sufficient, the effectiveness of using personas is greatly diminished. Furthermore, attempting a scaled-back or low-budget version of personas can cause more harm than good. If the research and analysis is not thorough, confidence in the resulting personas will be undermined and as identified in the study, if the persona details and goals are not correct then the design direction can be inappropriate. Using personas also requires training at an organisational level, this is an additional obstacle and given the lack of published material on how to use personas, can be difficult to overcome.

The debate about whether personas work or not has been one of faith versus scepticism; claim versus counter-claim. This study demonstrates the effectiveness of using personas in the product design process and while more research is needed, there is now some objective evidence that using personas does work.

References

Chapman, C.N, and Milham, R. P (2006) 'The persona's new clothes: methodological and practical arguments against a popular method' Proceedings of the Human Factors and Ergonomics Society 50th Annual Meeting, pp. 634 –636. (Available online at: http://cnchapman.files.wordpress.com/2007/03/chapman-milham-personas-hfes2006-0139-0330.pdf).

Cooper, A (1999) 'The Inmates are Running the Asylum', New York: Sams Publishing.

Cooper, A and Reimann, R. (2003) 'About Face: The Essentials of Interaction Design', Indianapolis: Wiley.

Ji-Ye Mao, Vredenburg, K, Smith, P. W, & Carey, T (2005) 'The state of User-Centred Design Practice', Communications of the ACM, Vol. 48, No. 3, pp 105-109.

McCloud, S (1994). Understanding Comics: The Invisible Art. New York: Harper Perennial.

Moraveji, N, Li, J, Ding, J, O’Kelley, P and Woolf, S. (2007) 'Comicboarding: Using Comics as Proxies for Participatory Design with Children', Proceedings from CHI 2007, April 28-May 3, San Jose, California, ACM: pp 1371 – 1374.

Mulder, S (2007) 'The User is Always Right; A Practical guide to using Personas for the Web', California: New Riders.

Nielsen, J, (1993) Usability Engineering, Boston: Boston Academic Press.

Nielsen, J and Landauer, Thomas K (1993) 'A mathematical model of the finding of usability problems' Proceedings of INTERCHI'93 Conference, 24-29 April 1993,Amsterdam, The Netherlands: ACM pp. 206-213.

Nielsen, J, & Mack, R. L (Eds.), (1994). 'Usability Inspection Methods' New York: John Wiley & Sons.

Pruitt, J and Adlin, T(2006) 'The Persona Lifecycle, keeping people in mind throughout product design' San Francisco: Morgan Kaufman.

Pruitt, J and Grudin, J (2003) 'Personas: practice and theory', Proceedings of the conference on Designing for user experiences, June 06-07, San Francisco, ACM pp 1-15.

Citations should use the following reference: Long, F (2009) 'Real or Imaginary; The effectiveness of using personas in product design', Proceedings of the Irish Ergonomics Society Annual Conference, May 2009, pp1-10 Dublin.