Although it’s pretty easy to get students to perform competently as facilitators in usability tests, it seems to be a lot harder to get them to analyse the data that they gather in the tests.
I heard about this at the OzCHI conference in 2005, where Mikael Skov and Jan Stage presented a paper about their work at Aalborg University. They teach HCI, and they think about how to teach it better.
- Mikael Skov and Jan Stage’s paper in the ACM Digital LIbrary: Supporting problem identification in usability evaluations
- The paper is also available as a .pdf (Skov and Stage 2005)
The paper looks at students analysing data from usability tests
They noticed this problem about data analysis and they tried a small-scale experiment.
They got a tape of a typical usability test. Each of the students who participated in their experiment reviewed the tape and recorded the usability problems. Eight of the students did this review without being taught how to analyse data. Six of the students did their review after being taught the use of a simple ‘conceptual tool’.
The students who had been taught the use of the conceptual tool found more problems and were more consistent in what they found.
Mikael Skov, in the modest way of academics, pointed out a few deficiencies in their research. For example, they did not consider the capture of positive findings. The groups were quite small. But still, the results are worth considering. I’m sufficiently convinced by Skov and Stage’s work to take the view that thinking about severity will help students to find more problems and be more accurate in their analysis of those problems.
The analysis “tool” expands on three severity levels
So what’s in the magic ‘conceptual tool’ that made so much difference?
It’s a matrix that compares severity (critical, serious and cosmetic) with four different aspects of experience:
- Slowed down (relative to normal work speed)
- Understanding
- Frustration
- Test monitor (that is, whether the participant in the usability test needed help from the person running the test).
So, for example, a ‘serious’ problem is characterised as:
- Slowed down: delayed for several seconds
- Understanding: does not understand how specific functionality operates or is activated. Cannot explain the functioning of the system.
- Frustration: is clearly annoyed by something that cannot be done or remembered or something illogical that you must do. Believes has damaged something.
- Test monitor: receives a hint.
I’ve put the matrix at the end of this article.
Teach severity scales and their impact as part of analysis
Now, those of us who do this stuff for a living might argue somewhat with aspects of this conceptual tool.
For example, I might look for a ‘minor’ category to mop up problems that are a bit more important than ‘cosmetic’ but don’t quite make it into ‘serious’. And I might argue that you need to be slowed down for a bit longer than ‘several seconds’ before a problem becomes ‘serious’. I also noticed that there are some gaps – is it really not possible to consider that something critical might cause frustration?
But I think the big lesson here isn’t the precise details of the tool. I think it’s the importance of teaching beginners an explicit, described way of categorising problems before they try looking for problems. And up to now, I’ve been doing it the other way around: getting them to look for problems, then asking them to determine how severe the problems might be.
I’d probably call it a ‘severity scale’ rather than a ‘conceptual tool’, but I’d aim to include both the classification of the problem and Skov and Stage’s secondary dimensions, which I’ll probably call something like ‘impact’ (suggestions, please?).
This article first appeared in Usability News, the online magazine of the British Computer Society’s HCI group.
Conceptual tool for problem identification
From (Skov and Stage 2005) available as a .pdf
| Slowed down
relative to normal work speed |
Understanding | Frustration | Test monitor | ||
|---|---|---|---|---|---|
| Critical | Hindered in solving the task | Does not understand how information in the system can be used for solving a task.
Repeats the same information in different parts of the system. |
Receives substantial assistance (could not have solved the task without it). |
||
| Serious | Delayed for several seconds | Does not understand how a specific functionality operates or is activated.
Cannot explain the functioning of the system. |
Is clearly annoyed by something that cannot be done or remembered or something illogical that you must do.Believes he has damaged something. |
Receives a hint. | |
| Cosmetic | Delayed for a few seconds | Does actions without being able to explain why (you just have to do it). |
Is asked a question that makes him come up with the solution |
#usability