Why do people fill in surveys?
Did you answer: “Because they’re hoping to win the prize in the draw that’s offered”?
No? I thought not. And of course, you’re right, but there is some evidence that guaranteed incentives can work.
I’m going to talk a little about the generalities, and then discuss an article in the February 2011 issue of Survey practice: The Effectiveness of a Monetary Incentive on Response Rates in a Survey of Recent U.S. Veterans by Steven S Coughlin, Pablo Aliaga, Shannon Barth, Stephanie Eber, Jessica Maillard, Clare Mahan, Han K. Kang, Aaron Schneiderman, Samar DeBakey, Patricia Vanderwolf, and Meredith Williams.
Response depends on trust, effort, and reward
If you’re familiar with Dillman et al (my survey book of the month for January 2011), then you’ll know that response depends on a mixture of factors:
- Trust: do they trust your organisation and what you’ll do with the data?
- Effort: how much time and mental effort do they expect this survey to take?
- Reward: what will they get from filling in the survey?
A lot of ‘reward’ comes from feeling like a good citizen, doing a favour for a respected organisation; intangibles like that.
But some ‘reward’ can come from a simple incentive such as cash. Dillman and his colleagues point out that putting a $1 bill in the envelope with a mail survey shows a level of trust and offers an immediate reward: both are benefits that can increase response rate.
Why don’t prize draws work all that well as incentives? Because of:
- Trust: people aren’t convinced the prize will really be awarded.
- Effort: a draw, especially a very valuable draw, can make it seem like the survey will be harder work.
- Reward: even if people believe the prize will be awarded, most of them don’t expect to win so there is no perceived reward.
Evidence confirms that immediate payments can help
The February 2011 article on incentives is a report on an experiment that looked at a survey of US veterans. They pointed out:
“Veterans may be reluctant to participate in a government survey because of concerns about divulging private information about sensitive topics. Anecdotal information suggests that some veterans may be reluctant to participate in health surveys because they don’t wish to have their government benefits or security clearances adversely affected. A further issue is that veterans are often invited to participate in health surveys and some veterans may be experiencing “survey fatigue.” On the other hand, veterans may wish to participate in a survey on veteran health topics because they believe the topic is important or they are altruistic”.
The Effectiveness of a Monetary Incentive on Response Rates in a Survey of Recent U.S. Veterans
See how the three factors come in?
- Trust (will they cut my benefits?),
- Effort (it’s a lot of work doing all these surveys) and
- Reward (I want to help with this important topic).
The researchers sent the same survey to balanced probability samples of US veterans. Some of them got just a mailed survey; some got a $5 bill with the survey; some were promised $5 reward for filling in the survey (and did indeed receive it). Respondents could send back a completed paper survey or do an equivalent survey online.
Summary of their results:
- Without any incentive, the survey achieved 21% response rate
- A guaranteed reward of $5 after the survey increased their response rate to 28%
- Enclosing $5 with the survey increased their response rate to 30%.
Is an improved response rate a good idea?
Those higher response rates are tempting. Generally, more response = better data. Why? Because the lower the response rate, the more ways in which the population that doesn’t respond can be different from the population that does respond in a way that affects the results of the survey (“non-response bias”).
These researchers also looked at whether the incentives changed the type of people who responded – and found differences:
“Across all 3 incentive groups, participants were more likely to be older and to have a higher level of education as compared with the overall sample.”
The Effectiveness of a Monetary Incentive on Response Rates in a Survey of Recent U.S. Veterans
Does that matter? I don’t know. It all depends on the purpose of the survey, and whether that purpose is likely to be affected by those changes or not.
For a UX survey, consider some immediate financial reward
What about our surveys in user experience? We don’t often use mail surveys, so enclosing a $5 in the envelope isn’t really a possibility.
But I wondered about whether it might be practical to offer each person who submits the survey a guaranteed small reward, such as a $5 voucher for a popular shopping site. Your views? Have you tried that?