In the chapter on Reports, I look at what you’ve learned from the data, what results you’ll put into your report, and its format. Our main tentacles in this chapter are ‘the answers you’ll use’ and ‘the reason you’re doing the survey’.
The topics in the chapter are:
- Think about what you learned, numerically
- Decide what news to deliver and when
- Decide what format to use for delivery
- Choose ‘inverted pyramid’ for most presentations
- There are many ways of showing the same results
- The best insights come from using surveys alongside other methods.
The error associated with this chapter is Total Survey Error – the consequence of all the individual errors you may have made along the way.
I couldn’t give all the appropriate origins and suggestions for further reading in the chapter, so I’ve collected them here. If there is something else you would like then please let me know.
10 principles to use for a calculated statistic
If you prefer to have a checklist for exactly what to do to report statistics accurately, then this one from Steel, Liermann et al (Beyond Calculations: A Course in Statistical Thinking, American Statistician 2019) is a great help:
1. Plot your data—early and often.
2. Understand your dataset as one of many possible sets of data that could have been observed.
3. Understand the context of your dataset—what is the background science and how were measurements taken.
4. Be thoughtful in choosing summary metrics.
5. Decide early which parts of your analysis are exploratory versus confirmatory and pre-register your hypotheses in your own mind.
6. If you are going to use p-values, which can be useful summaries when testing hypotheses, follow these principles:
- Report estimates and confidence intervals;
- Report the number of tests you conduct (formal and informal);
- Interpret the p-value in light of your sample size (and power);
- Don’t use p-values to claim that the null hypothesis of no difference is true;
- Consider the p-value as one source of support for your conclusion not the conclusion itself.
7. Compute (and display) effect sizes and confidence intervals as an alternative to or in addition to statistical testing.
8. Consider creating customized, simulation-based statistical tests for answering your specific question with your particular dataset.
9. Use simulations to understand the performance of your statistical plan on datasets like yours and to test various assumptions.
10. Read with skepticism, remembering that pattern can easily occur by chance (especially with small samples), and that unexpected results based on small sample sizes are often wrong.
Recognise what different stakeholder audiences need
Janet Six onced asked me about tailoring UX presentations to different audiences, specifically development teams and executive teams. My view is that audiences are likely to share some concerns, but that they are mostly interested in what they specifically are being asked to do in future: More details here:
I was convinced by evidence to try assertion/evidence reports
In my book, I urge you to change your presentation style to ‘assertion/evidence’ – where each slide has a title that is a full sentence containing the main point of the slide.
The paper that convinced me to change to ‘assertion evidence’ was Alley, M., & Neeley, K. A. (2005). Rethinking the design of presentation slides: A case for sentence headlines and visual evidence. Technical communication, 52(4), 417-426. At the time of writing this page, a .pdf of this paper is openly available: Alley and Neely 2005.
More recently, Garney and Alley worked with engineering students to compare topic/subtopic (the traditional style of presentation slides) with assertion/evidence and found that assertion/evidence helped with understanding and recall. Garner, J., & Alley, M. (2013). How the design of presentation slides affects audience comprehension: A case for the assertion-evidence approach. International Journal of Engineering Education, 29(6), 1564-1579. At the time of writing this page, a .pdf of this paper is openly available: Garner and Alley 2013
If you’d like to read even more about it, then try Michael Alley The Craft of Scientific Presentations: Critical Steps to Succeed and Critical Errors to Avoid (Springer 2013). It has plenty of useful tips and advice.
It took me a while to get used to the assertion/evidence style, but now I would not go back. And I also realised that I could write headings in documents as mini-sentences – a style I also urge you to try. I do not know of any study that compares sentences in documents to traditional shorter topic headings.
Write about percentages using descriptive language
Some people find percentages such as “52%” or “17%” easy to understand and think about. For others, this is hard to grasp and they would be prefer a more descriptive term such as “about half” for 52% or “under a quarter” for 17%. To meet both needs, you can write like this:
- Lead with the number and follow up with the description “52% (about half)”, or
- Lead with the description and follow up with the number “about half (52%)”
Zachary Grimshaw told me about a handy chart published by The Eastern Institute of Technology in New Zealand that displays descriptive langauge for writing about percentages (.pdf).