Why do people fill in surveys?
Did you answer: “Because they’re hoping to win the prize in the draw that’s offered?”
No? I thought not. And of course, you’re right, but there is some evidence that incentives can work. I’m going to talk a little about the generalities, and then discuss an article in the February 2011 issue of Survey practice.
Response depends on trust, effort, and reward
If you’re familiar with Dillman et al (my survey book of the month for January 2011), then you’ll know that response depends on a mixture of factors:
- Trust: do they trust your organisation and what you’ll do with the data?
- Effort: how much time and mental effort do they expect this survey to take?
- Reward: what will they get from fillling in the survey?
A lot of ‘reward’ comes from feeling like a good citizen, doing a favour for a respected organisation; intangibles like that.
But some ‘reward’ can come from a simple incentive such as cash. Dillman and his colleagues point out that putting a $1 bill in the envelope with a mail survey shows a level of trust and offers an immediate reward: both are benefits that can increase response rate.
Why don’t prize draws work all that well as incentives? Because of:
- Trust: people aren’t convinced the prize will really be awarded
- Effort: a draw, especially a very valuable draw, can make it seem like the survey will be harder work
- Reward: even if people believe the prize will be awarded, most of them don’t expect to win so there is no perceived reward.
New evidence confirms that immediate payments can help
The February 2011 issue of Survey practice includes an article by Coughlin, Steven S., Pablo Aliaga, Shannon Barth, Stephanie Eber, Jessica Maillard, Clare M. Mahan, Han K. Kang, Aaron Schneiderman, Samar DeBakey, Patricia Vanderwolf, and Meredith Williams on The Effectiveness of a Monetary Incentive on Response Rates in a Survey of Recent U.S. Veterans (2011).
Their experiment looked at a survey of U.S. veterans. They pointed out: “Veterans may be reluctant to participate in a government survey because of concerns about divulging private information about sensitive topics. Anecdotal information suggests that some veterans may be reluctant to participate in health surveys because they don’t wish to have their government benefits or security clearances adversely affected. A further issue is that veterans are often invited to participate in health surveys and some veterans may be experiencing “survey fatigue.” On the other hand, veterans may wish to participate in a survey on veteran health topics because they believe the topic is important or they are altruistic”.
See how the three factors come in? Trust (will they cut my benefits?), effort (it’s a lot of work doing all these surveys) and reward (I want to help with this important topic).
The researchers sent the same survey to balanced probability samples of US veterans. Some of them got just a mailed survey; some got a $5 bill with the survey; some were promised $5 reward for filling in the survey (and did indeed receive it). Respondents could send back a completed paper survey or do an equivalent survey online.
Summary of their results:
- offering a guaranteed reward of $5 increased their response rate by over 30%
- enclosing $5 with the survey increased their response rate by over 50%.
Is an improved response rate a good idea?
Those higher response rates are tempting. Generally, more response = better data. Why? Because the lower the response rate, the more ways in which the population that don’t respond can be importantly different from the population that do respond (“non-response bias”).
These researchers also looked at whether the incentives changed the type of people who responded – and found differences: “Across all 3 incentive groups, participants were more likely to be older and to have a higher level of education as compared with the overall sample. ”
Does that matter? I don’t know. It all depends on the purpose of the survey, and whether that purpose is likely to be affected by those changes or not.
Lessons for ux surveys
What about our surveys in user experience? We don’t often use mail surveys, so enclosing a $5 in the envelope isn’t really a possibility.
But I wondered about whether it might be practical to offer each person who submits the survey a guaranteed small reward, such as a $5 Amazon voucher. Your views? Have you tried that?
featured image ‘Survey’ by Sean MacEntee, creative commons licence
#surveys #surveysthatwork