Bonus content: Choosing a Standard Questionnaire such as SUS of Forsee

As we’ve seen, building a questionnaire from scratch – especially one that includes a Likert scale – involves many processes and pitfalls. In that case, why not use one of the standard questionnaires?   

Pros and cons of using an existing questionnaire

An existing questionnaire has three clear advantages: 

  1. Someone else has done some of the hard work of question design.  
  2. Some of the best-known standard questionnaires have been thoroughly tested and validated for you.
  3. And, crucially  You may be able to compare your results with others who are using the same standard questionnaire. 

The disadvantages are that: 

  • It may not be a perfect match for your goals and therefore may not include your Most Crucial Question. 
  • The questionnaire may not capture the Burning Issue. 
  • People in the group that you want to answer may not understand the questions in the way the designers originally intended. 

The Oxford Hip is an example of a standard medical questionnaire 

As I write this, I’ve got an ‘Oxford Hip’ on my desk from my orthopaedic surgeon. I had a successful hip replacement a few years ago, and every five years I’m asked to complete this questionnaire as part of the follow-up. It’s a Likert scale and includes items about everyday life such as walking and getting dressed.  

 image missing here

165  A question from the Oxford Hip medical assessment questionnaire 

If you wondered whether the steps I mentioned in making a Likert scale are excessive, have a look at the development of this questionnaire. (Dawson, Fitzpatrick et al. 1996). That’s exactly what I mean when I say “thoroughly tested and validated”, and it’s why my surgeon relies on it.  

I’m guessing, though, that you’re not looking at medical measurement, so let’s think about a couple of questionnaires that are more widely used in business in general.  

The System Usability Scale, SUS, has 10 statements for people to rate 

Created back in the pre-internet days of 1986 by John Brooke, SUS features 10 statements about the usability of a system or product. People are asked to judge the extent to which they agree or disagree with these on a five point scale. Combining those answers using the SUS method will give a score between 0-100. 

If you choose to use SUS make sure your survey goals are about assessing usability. In common with another of the standard questionnaires I feature in the book – Net Promoter Score – SUS is not a diagnostic tool: the results won’t shed much light on why people have answered as they have.  

 image missing here

166 The System Usability Scale SUS 

Digital Equipment Corporation makes SUS free to anyone to use, provided they cite the copyright notice.  

If measuring comparability is one of your goals SUS is a strong contender

Standard formats such as Net Promoter Score and SUS come into their own when your goal is to compare how your system is performing with those of competitors or those working in the same field.   

For example, you might choose SUS for these goals: 

  • tracking how your system or product is performing,  
  • exploring trends or, 
  • working out the effectiveness of any interventions you made as a result of previous surveys.  

But you can only compare your results from SUS with other results if you keep the questionnaire exactly the same – or if you do change it, then you have a lot of work to do to check that your new questionnaire has not affected the scores. 

For example, look at question 8 on the SUS (“I found the system very cumbersome to use”). The word ‘cumbersome’ is not well known, and if the people who want to answer don’t know the word then they won’t know how to answer the question. 

Stepping back a bit: think about the word ‘system’. If you’re measuring a website or web app, does ‘system’ work for you?  

If you change the questionnaire, you lose comparability 

When you start with a standard questionnaire, it’s very tempting to think “well, I’ll change it to work better for my goals and the people in my defined group.” 

Unfortunately, creating a hybrid comes at the cost of a trade-off: more relevance to the people you are surveying perhaps but less or even no comparability. It also may not be as good as using the original questionnaire.  

For example, let’s look again at SUS. In their study ‘A Comparison of Questionnaires for Assessing Web Usability (2006), Tullis and Stetson compared SUS with four more recent variants – including one they developed themselves – and discovered that the original SUS worked better at identifying which of two websites was easier to use, and at smaller sample sizes.   

More recently, Jeff Sauro and Jim Lewis have done lots of statistical work and testing on other versions of SUS, including changing the statements so that they are all worded positively and work better for websites. Have a look at www.measuringu.com) for their work.  

The Foresee proprietary questionnaire is popular 

SUS is free to use by anyone who is willing to cite the copyright notice. At the other end of the scale, many market research organisations offer proprietary services that aim to measure topics that are popular with managers, such as (you’ve guessed it), satisfaction. 

Let’s have a quick look at one of these: Foresee from Forsee.com. Here’s an extract from one of their typical questionnaires.  

 image missing here

167 The first two questions from a Foresee questionnaire that I received myself 

I’m going to hand over here to Erica Hall. Here’s what she said in her article “On Surveys (2015)”  

There’s a whole industry based on customer satisfaction. And when there is an industry that makes money from the existence of a metric, that makes me skeptical of a metric.  

These are the questions that sounded good to ask, and that seem to map to best practices. 

But this is complete hogwash. 

Rate the options available for navigating? What does that mean? What actual business success metric does that map to? Rate the number of clicks–on a ten point scale? I couldn’t do that. I suspect many people choose the number of clicks they remember rather than a rating. 

 None of this speaks to what the website is actually for or how actual humans think or make decisions. 

And, most importantly, the sleight of hand here is that these customer satisfaction questions are qualitative questions presented in a quantitative style. This is some customer research alchemy right here. So, you are counting on the uncountable while the folks selling these surveys are counting their money. Enjoy your phlogiston. 

(Hall 2015) 

Questions to ask before using a standard questionnaire

Three questions to ask about using a standard questionnaire 

If you’re considering a standard questionnaire, ask yourself these questions: 

  1. Do these questions make sense for our goals? 
  2. Do these questions make sense for the people that we want to ask? 
  3. Is comparability important for our goals?