Doing research with people who are not users: consultation

Subject matter expert - part of a doodle by Naintara LandA woman with lots of curly hair is wearing a top with a namebadge 'SME' and a pleated skirt. She carries a pile of papers with the title 'Expert handbook'
We have a learned a lot by working with subject matter experts.

This post is co-authored by Naintara (Tara) Land, formerly Head of User Research for the UK Government Digital Service (the people behind GOV.UK).

User research is central to our work in User Experience: doing research to find out about our users, then acting on what we’ve learned—or persuading our colleagues to act on a shared understanding of what we have learned.

But what about doing research with other people?

We admit that, in the past, we’ve often battled with managers who believe that consultation is all that matters and can replace user research. We’ve lost count of the number of conversations we’ve had that went like this:

User researcher: “We need a budget for recruiting users to be research participants.”

Manager: “Why? We have Subject Matter Experts (SMEs) on the team who have years of experience using this product. Why can’t you rely on them for the usability testing?”

Both of us have worked a lot for the UK government, for which a key method of gathering information about a proposed new thing is to publish a consultation paper. The purpose of a consultation is to obtain the views of invited special-interest groups and, in theory at least, interested members of the general public. So another conversation we’ve had too many times goes like this:

User researcher: “We need to do some research with real users.”

Manager: “Why? We’ve received dozens of comments on the recent consultation paper.”

Of course, we understand that part of our role is to explain why we need to do user research throughout a project. However, more recently, we’ve come to see that there is a place for consultation alongside user research. For example, consultation can get you up to speed on a new subject area more quickly. Plus, it can help you identify edge cases and scenarios that you need to accommodate in ways that would be hard to identify through conventional user research. So don’t confuse consultation with user research, but do use both approaches.

We’re going to explain:

  • what we mean by consultation,
  • why it’s different from user research, and
  • how we think it can add value.

Consultation involves people who aren’t users

One of the key differences between user research and consultation is who you involve in a research activity.

When we do user research, we work hard to ensure that we find participants who belong to the exact target audience for the product or service we’re working on. We use the term consultation for any type of research with anyone else. For example, a senior manager decides to try using the product your team is working on. Might that manager be part of the actual target audience? Maybe so, if the product is for busy professionals. But the manager might be nothing at all like an actual user—for example, if the product is for teenagers who are trying to get their first job. In either case, the senior manager’s views are well worth paying attention to, but this type of consultation is not user research.

Consultations typically get input from two sorts of people:

  • subject-matter experts (SMEs): often their expertise comes from previous experience as users, but they do not currently belong to the actual target audience.
  • user advocates: these are people from charities, professional organisations, or pressure groups who represent  specific point of view or people with particular needs.

Many product teams correctly place a lot of value on having people working on their team as SMEs, whether occasionally or permanently, who have first-hand knowledge and experience of being users, as illustrated in Figure 1.

A large group of people, may of them women with curly hair wearing white tops and pleated skirts, are labelled as 'users'. One of the women has stepped into a spotlight and is labelled as 'SME'. She is carrying a pile of papers with the title 'Expert Handbook'.
Figure 1—Many SMEs have extensive experience as users (Image credit: Naintara Land)

Some people—especially within the disability rights movement—argue that a team cannot be truly user-centred unless it includes people who belong to the exact target audience: the slogan “Nothing about us without us” captures this idea.

We’ve also chosen to consider user advocates because of the fascinating work they do around raising awareness of and creating understanding of the needs of the groups they represent. For example, if you wanted to understand the impact of a new piece of legislation on people in the UK who are deaf or have hearing loss, it would be a good idea to ask RNID, the UK National Hearing Loss charity, for their views.

Clearly, there are many other variations on these themes. The senior manager we mentioned earlier undoubtedly has valuable subject-matter expertise. SMEs may have been user advocates in the past or continue to combine user-advocacy activities with their current role.

And there’s a variation of this that often applies to us personally, in our work as UX consultants: negotiating the overlaps between our role as consultants—whether as internal employees or outside experts—and consultation, by which we mean providing our knowledge and expertise. Plus, sometimes, to complicate matters further, we happen to actually belong to the target audience for the product or service, so are also users. We deal with this variation by making sure that we frequently ask ourselves, “Which role am I playing right now?”

User research aims to discover what users do

Our overall aim for user research is to understand what users do and why they do it. It helps to inform us about user needs, then supports the design process by providing feedback about whether the product or service is meeting those needs.

So, when we do user research, we mostly choose methods that gather behavioural data to find out what users do. We conduct contextual interviews with users to understand how their environment and goals influence what they do. We do card sorting to establish how they group topics or where they would look for them. And, of course, we do lots of usability testing, watching users try to use a product or service.

Consultation aims to discover what people know

Our overall aim for consultation is to discover what people know about a product or service, a business, or organisational needs. We might conduct a workshop with stakeholders to identify key priorities and areas of interest and concern. We may have one-on-one meetings with SMEs to learn about the history of the product or service—focusing on successes and failures so we can try to avoid repeating old mistakes. Or we might have a public meeting or call for interested parties to comment on proposed changes.

Consultation differs from user research

In the UK, many government services use consultations to call for comments on proposed changes. The UK Government Digital Service also mandates user research as part of the process for creating government services. In 2015, the UK Ministry of Justice described the results of a consultation on proposed changes to court fees. They asked questions about the details of some proposed changes and learned about respondents’ knowledge of the system, as well as their views on how the changes might work.

Alongside that, user researchers at the Ministry of Justice described how they conducted user research to improve their service that helps people with their court fees. They did usability testing of form prototypes with typical users. They also looked for behavioural data to establish whether the proposed new forms really did make it easier for users to get through the necessary processes.

Consultation can help you improve the user experience

If you followed the links to investigate the Ministry of Justice consultation, you discovered that the reports are full of very technical language and detailed questions. You may be surprised that we think consultation can improve the user experience of your product or service, but we do, and that’s what we will focus on for the rest of this column.

SMEs help you understand the domain much more quickly

SMEs can help you understand the history, terminology, and aims of your product or service, bringing you up to speed much more quickly. The clue is in their name—subject-matter experts know about, well, the subject. They often have many years of experience in a domain and have previously seen teams try out all sorts of good and bad ideas. Consulting an SME appropriately can enable you to understand important issues such as legislation and standards—and avoid rookie errors in the scoping and design of your product or service. Of course, sometimes what was a bad idea some years ago might actually be a good idea today. But it’s helpful to at least have some knowledge about the past to avoid repeating an old mistake in exactly the same way, with the same result.
The knowledge and context that you can gain through consultation with SMEs can help you make better use of your time during user research. Instead of hearing about the domain for the first time from your users, you’ll just be validating what you already know and extending that knowledge.

For example, when Caroline was working on a system for farmers in the UK, she found it tremendously helpful to have discussions with an SME who explained many of the technical terms the farmers used in that specific context. For example, the term feature did not mean something the system could do, but instead referred to a permanent structure on the land—such as a water tower. The SME also explained some sensitivities about an earlier version of the system that helped her to prepare for negative reactions to the proposed new system from some users during her subsequent user research.

Referring back to our Ministry of Justice example, a new user researcher working in the area of fees might find it helpful to know that the courts use the term fee remission to mean a reduction in the court fee for a person on a low income.

Advocate groups help you learn about hard-to-reach groups

Another important advantage of consultation is that user-advocate groups have a more comprehensive understanding of niche or specialist use cases that might not come up during typical user research, but would have a significant influence on the overall viability of the product or service you are designing. An example: the UK suicide-prevention charity, Samaritans launched an app called Samaritans Radar, which they described as:

“A free Web application that monitors your friends’ Tweets, alerting you if it spots anyone who may be struggling to cope. The app gives users a second chance to see potentially worrying Tweets, which might have otherwise been missed.”

Although the original press release for the app mentioned consultation with academic experts, there was an outcry about the potential for those with less than wholesome intentions—such as stalkers, trolls or bullies—to exploit the app. The Samaritans recalled the app nine days after its launch. While we expect that concerns over privacy might well have emerged eventually through user research, perhaps the particular threat that stalkers and bullies posed could have been identified more quickly through consultation with other organisations that support vulnerable people.

Looking again at the Ministry of Justice example, one of the questions that came up during the consultation was specifically about the impact of changes to fees on “court users who have protected characteristics.” In the UK, the Equality Act 2010 uses the term protected characteristics to refer collectively to nine areas in which discrimination is illegal: disability, gender reassignment; marriage and civil partnership; pregnancy and maternity; race; religion or belief; sex, and sexual orientation.

Using the sort of participant recruitment that is typical of user research, it might have been difficult to find someone on a low income who needs help with their court fees and is also willing to disclose their sexual orientation in court. Reaching out to some LBGT+ advocacy groups might be a good way to discover specific concerns of people in those circumstances and would help the team to decide how best to involve them in user research.

Consultation can be misleading

The term remission neatly illustrates one example of how consultation can be misleading. Once again, the clue is in the name: SMEs are experts and, therefore, are immersed in the language of their area of expertise. The Ministry of Justice user researchers found that people on low incomes did not realize what remission meant and were confused about the difference between legal aid (money from the government to pay for a lawyer) and fee remission (help with the fees they had to pay to the court).

We’ve found that people joining our teams as SMEs become full-fledged team members really quickly, which is wonderful. But it also means they rapidly become familiar with our jargon and the concepts for the new service. For example, if you were a member of the team that came up with the idea of filleted accounts, you’d probably struggle to understand how odd this term sounded to us when we heard it for the first time.  (In case you are curious, the blog post announcing the relevant change, Accounts advice for small companies, explains: “Companies that don’t opt to file their director’s report and profit and loss are said to be filing “filleted” accounts”.)

Similarly, based on their experience, SMEs may be able to tell you in great detail and depth how to do things. But the places they’ve worked or their contexts of use naturally limit and constrain their perspective rather than encompassing all variations and scenarios. Therefore, relying solely on SMEs for your understanding of user needs may introduce bias into your designs. It’s also possible that bias may occur just because you consulted or chose SMEs to work on a project specifically because their experience and consequent viewpoints might be more favourable to or aligned with the agenda of other stakeholders on the project.

For example, Tara recalls a teacher who acted as the SME on a project focusing on how to provide curriculum based–learning resources online. The teacher explained that that lesson preparation included creating both schemes of work and lesson plans. The team, including the SME, then embarked on a month-long discovery effort, carrying out contextual user research in over 18 schools across the country.

During discovery, the team collected artifacts, including lesson plans—ranging from illegible scribbles in a pocket diary, at one end of the spectrum, to Word documents that detailed the learning objectives, starter, main and plenary portions of the lesson, as well as how to adapt the lesson for less-able and higher-ability students. This provided a very tangible demonstration of the range and diversity of planning artifacts. But, while the SME was able to point the way, there was still a need for user research to enable the team to understand in detail how to do things.

SMEs and advocates who are also users may not be typical

The final pitfall we’ve encountered is confusing the user with the typical user. We’ve seen projects on which user research was conducted solely with SMEs who were also users. Maybe, when teams recruit users internally, SMEs are more likely to sign up. Maybe the project team worked hard to bring a user onto the team to act as an SME, then couldn’t get the budget to bring in any other users for research.

The risk is again the problem of expertise. When working solely with SMEs, you’re unlikely to capture the needs of the full spectrum of users. What about new recruits who have never seen the product before? What about the users who use the service only very occasionally, while the SME uses it every day? Maybe the SME has excellent digital skills, while some users may have low digital skills. User research with a full range of users would fill in those gaps and help you to create a more robust design, ensuring that everyone can use your product or service.

Similarly, we’ve seen challenges arise when a particular advocacy group captured stakeholders’ attention to the exclusion of other people who do not share their point of view. For example, one service became fixated on the needs of people with deep concerns about privacy and data sharing—even though the majority of users did not care about this. (Maybe the users ought to have been more worried about these issues, but the fact was that they weren’t.)

Consultation saves time and can deliver better value from user research

You need to do both user research and consultation to resolve the tension between designing for the lowest common denominator, in terms of skill and knowledge, and supporting niche scenarios and edge cases. Consultation can save you time and money by flagging issues before you’ve committed too much effort to designing a solution that may just not be viable.