Chapter 4 Questionnaire: Build and test the questionnaire

In this chapter I’ve written about why grids in questionnaires may be a problem. It’s a topic I studied for the organisation Kantar, who were concerned about the 15% of respondents who break off from questionnaires because of large grids. Clearly one way of solving the problem is: don’t use grids. But for various excellent operational reasons, many Kantar questionnaires must deal with a set of questions that are a series of rating responses – usually provided as a grid, such as the example in 3 A  from (Conrad, McCullough et al. 2014). 

 insert image here

3 A set of rating responses arranged as a conventional grid 

We worked to create a new response format, the ‘dynamic grid’. I remember the initial workshop where someone – I wish it had been me! – said ‘why not try something like iTunes Cover Flow?’  

A few sketches later and we had the core of our new format – called a ‘dynamic grid’ by Kantar because of the way each new topic appears on a card when the person who answers chooses their response to the previous card.  

 insert image here

4 A set of rating responses arranged as a dynamic (paging) grid with context 

The workshop part was quick and fun. It then took a long, long time to get through the necessary tests including: 

  • usability tests with respondents 
  • expert reviews with questionnaire professionals
  • pilot tests in the field 
  • a/b experiments conducted by survey methodologists, such as the one reported in (Conrad, McCullough et al. 2014). 

Here’s one example of the details to investigate: is it better to show hints of text on the previous and next tile (“context”), as above, or might they be distracting? And is the dynamic grid better than the ordinary type? 

 insert image here

5 A set of rating responses arranged as a dynamic (paging) grid without context 

The a/b experiment that those images came from compared the two types of dynamic grid with the conventional grid on short (6 statements), medium (14 statements) and long (28 statements) tasks. Remember that the whole idea was about reducing break-offs from that nasty 15%?  

 insert image here

6 The dynamic grids had significantly fewer breakoffs for medium tasks; no difference for short or long tasks 

The results of the a/b test are mixed: our new format could help with a medium length task but didn’t make any difference for short or long tasks. It was a lot of work for a modest gain for a highly specific length task. 

The morals of that story are:  

  1. Don’t assume that your fun, fancy interaction method will work once it gets stuck into a questionnaire. It may be great, or it might simply increase your break-off rate. Or it may not make enough of a difference to justify the effort you put in.
  2. You’ve got to test, test, and test again.   

While you’re at it, you might want to think twice about using sliders

Here are two useful articles on using sliders for your rating scale: