For this month’s post for my Surveys that Work blog on Rosenfeld Media I’m handing over to Gerry Gaffney – for a forensic examination of one customer survey he recently encountered. Apart from being co-author of my book, Forms that Work, Gerry is an expert in user experience research and design. His company is Information and Design.
Recently I stayed at a Radisson hotel in Sydney for a few nights.
After I left, I was invited by email to respond to a survey about my experience. As these things go, it wasnât badly designed. Iâve listed a few specifics below, but itâs also worth making some general comments to consider when sending out this type of survey.
Respondents are self-selecting
Why did people respond to your survey? Did you over-sample those who were really happy, those who were really unhappy, or those who have time on their hands? I donât know how to offset this, other than by trying to get as high a response rate as possible. Anything that reduces this response rate may militate against minimising the self-selection.
In this case, Radisson didnât offer me any incentive to respond. Many busy people would automatically delete such invitations. As Caroline has pointed out previously on this blog (Do incentives help to improve response rate?), an incentive may help increase response. I confess that only my interest in forms design persuades me to respond to voluntary surveys from commercial organisations, but a $5 Amazon token would probably spark my acquisitiveness or loss-aversion bias.
Respondents are busy
Radisson promised that the survey would take â5 to 8 minutesâ (a remarkable degree of precision, I thought, and wondered how it had been achieved).
Respondents don’t care as much as you do
This is a perennial problem that UX people spend so much time communicating. Itâs often hard for teams to realise that in the real world âout thereâ, nobody cares very much about their product or service. This means that expecting deep thought and analysis from them is unrealistic.
Values and labels
Radisson asks respondents to rate various elements on a 10-point sale. Theyâve also added linguistic labels (âExcellentâ, âVery Goodâ and so on).
Possibly the labels are to help people interpret the numeric scale. If this is the case, then the numeric scale is inadequate, and should be redesigned.
The labels unfortunately present an unbalanced scale. At least three of the labels (âExcellentâ, âVery Goodâ, âGoodâ) are unambiguously positive, and only one (âPoorâ) is unambiguously negative. The lack of balance presents a bias â respondents have at least 6 places to assign a positive response, and a maximum of 4 to assign a negative one.
The lack of precision in the previous paragraph is a result of the ambiguous âFairâ. Some people may interpret this as a positive rating, some as a negative. For example, Elizabeth Buie in a tweet on this topic said that she interprets fair as âslightly negativeâ. This more or less aligns with my personal interpretation, although in other contexts âfairâ means âreasonableâ or âappropriateâ. In any case, the apparent attempt to clarify the purpose of the numeric score has resulted in a muddied solution.
Mid-points
Many survey designers seem to find neutrality anathema. The 10-point scale used by Radisson, if it is balanced, does not allow the respondent to be neutral. Iâve often heard people say things like âwe want them to make a choiceâ. In practice, when observing people filling in paper-based questionnaires, you will see people circle the two points on either side of the non-existent mid-point. If you demand that they make a single selection, they will choose one. However, what this achieves is to extract an incorrect response from someone who doesnât believe that response is appropriate. Itâs not exactly water-boarding, but it does amount to a forced confession whose value is tainted by the method of extraction.
Donât know, donât care, canât remember, and not applicable
Radisson wanted to know whether Iâd been asked to join some rewards program. I canât remember, but there is no option to tell them so.
In some cases (see below) they did allow for âN/Aâ as an answer. However, âN/Aâ is required much more frequently than many people realise. If in doubt, include such an option. However, consider whether people really understand what âN/Aâ means. Remember that itâs jargon, and ambiguous (not applicable, not available, no answerâŠ) .
Fatigue
At some point respondents get tired of your survey. I would hazard a guess that this occurs at a point way, way earlier than most organisations realise. In the case of the Radisson survey, they have done two things to mitigate this:
- Provided an up-front estimate of time to complete (the â5 to 8 minutesâ, remember?)
- Included a progress indicator at the top right.
Progress indicators are great; they can extract additional effort from the recalcitrant. However, their power is limited.
In this case, after several screens, Radisson will probably find that respondents are taking the path of least resistance, either abandoning the survey entirely or just clicking down a vertical line with very little attention to the individual questions theyâre answering.
I wonder whether any respondent really wants to rate both water pressure and drainage on a 10-point scale?
The preventative for fatigue is brevity.
UI design
 Radisson used check boxes instead of option (radio) buttons in one screen. Strangely, they went to the extra effort of creating an error state to trap when a respondent chose two options that were supposed to be mutually exclusive:
The message should have read âSorry. Please imagine that the check boxes are radio buttonsâ.
Wording
In usability testing, one of the first things you find out is when wording is confusing or, as in this example, unintentionally amusing. Radisson asks whether I âtookâ their breakfast. Not being a thief, of course I didnât.
If you donât want to know, donât ask
Finally, Radisson asks me if I experienced a problem, but allows me a paltry 30 characters to respond (if I donât choose one of their pre-defined options). Small, arbitrary limitations are rude and inconsiderate.
Iâve frequently had people justify such limits by saying âWe donât want them to send in âWar and Peaceâ.â Iâm not aware of any respondent who has actually attempted to do this, but it would certainly be reasonable to truncate anything after, say, two pages of text. But not 30 characters!
Itâs reasonable to alert people that space is somewhat limited, but âwordsâ rather than âcharactersâ should be the measure. In English, a word is around 5 letters, so you can divide your total by this number, add a little fudge factor, and list that. For example, if your field is restricted to 1024 characters, you can list this as â200 wordsâ. Only people like Caroline and me would ever count them and castigate you for it.
Conclusion
The Radisson survey is not an egregious example. It probably works quite well for the organisation. But a little more attention to detail, and a little usability testing, could make it better.
And thatâs almost universally true.