Chapter 1 of my book features a matrix to help you choose the right method for getting the information you need to make a decision. Here are a couple of questions you may have.
FAQ: Your matrix is all very well, but isn’t it over-simplified? You’ve left out lots of methods
Well, it’s definitely simplified. I’ve included the methods that I use most often, but I don’t want to put you off thinking about other choices too.
If you’d like a bigger matrix with a richer choice of methods, have a look at Christian Rohrer’s article When to Use Which User-Experience Research Methods.
He has a big matrix with lots of methods at the start of the article, and a smaller matrix that inspired my one.
FAQ: In your “route around the matrix” you haven’t got any quantitative/observe methods such as ‘analytics’ or ‘A/B’ tests. Why not?
For a Light Touch Survey, I find that I get better results if I focus more on iteration, using the mixture of methods in that route.
If it’s a Big Honkin’ Survey that you’ll use to make a big variety of different decisions, then definitely you’ll want to test it with plenty of A/B tests and other types of experiments. For some ideas about how to do it, have a look at the Working Papers pages of the US Bureau of Census: http://www.census.gov/library/working-papers.html
They do constant experiments because of the size and importance of the surveys that they run.
That’s typical of survey methodology as a profession. For example, in the introduction we had a look at (James and Bolstein 1992). They tested the effects of seven different types of incentive against no incentive.
If you’d like to keep up with the sorts of experiments being done by survey methodologists in general, then a good place to start is the AAPOR’s open access journal Survey Practice