Recently, I’ve been thinking a lot about errors in services, to what extent we know and measure them, and how that impacts both service users and our organisations.
As part of this I took part in a discussion with HMRC colleagues. We discussed all sorts of errors, and came up with five categories:
- Problems along the way
- Wrong result
- Unnecessary action
- Delayed-impact problem
- Non-uptake or over-uptake
A “problem along the way” is an error in the journey
We know that some people will start their journey on a service but fail to get to the end of it. The drop off is usually measured as ‘completion rate’ in government services, or described as ‘conversion rate’ in e-commerce.
“Problems along the way” (also known as ‘conversion problems’) include:
- not understanding a question
- the available responses for a question not including a choice that matches the person’s answer
- hitting error messages within the service
- struggling with uploading necessary supporting documents, or finding paper versions to attach to a paper form
- credential problems such as forgetting a password (online forms only)
- needing to contact the service for assistance in completing.
Yan Matusevich posted this example on Bluesky of having to restart a process five times.
The digitization of everything is becoming unbearable. My grandmother receives compensation as a Jewish victim of Nazi persecution from the German government. She received a letter instructing to her to confirm she’s still alive by a) downloading an app b) taking a selfie + photo of her documents.
It took me 30 minutes to figure out how to get the app to work, it crashed 5 times, [I] helped my poor grandmother re-take a selfie several times. She’s pretty tech-savvy for an 86-year-old, but this is just ridiculous. To add insult to injury, the online app is called “Survivor’s Portal” (!)
Yan Matusevich posting on Bluesky, January 2025
A “wrong result” happens when the outcome is incorrect
If you have ever tried to buy a product but something else got delivered, you have experienced a ‘wrong result’.
Sometimes that happens because of a problem along the way: I’m sure I’m not the only person who has mistakenly ordered one carrot when I intended to buy a kilo of them. But sometimes it’s because of a mistake made later, such as the time the supermarket decided to send me two lettuces instead of the two pork pies I ordered.
Occasionally, a wrong result can have more dramatic consequences such as the person who could not sell a car because of an error by someone at an insurer when typing a registration number.
In the world of government services, a wrong result can be that someone gets paid too much or too little of a benefit, or gets an incorrect decision. Some individual cases hit the headlines, such as a disabled woman who was asked to repay £5,000 of benefits despite being told repeatedly that she was getting the correct amount.
It is more difficult to find statistics about wrong results as a proportion of total cases. One of the few published examples is that of the immigration tribunal appeals. For example, from October to December 2024, “Almost half (43%) of the 6,700 cases determined at a hearing or on the papers were allowed/granted”, which means that the tribunal decided that the original outcome was wrong.
An “unnecessary action” is something that is extra
An unnecessary action is something that a person does because they do not know, or do not trust, what is happening based on their attempt to complete a service. The relevance to cost-per-transaction is that these unnecessary actions by people outside an organisation create costs for the organisation, such as the cost of dealing with a phone call or answering a letter or email.
Examples can be:
- checking that an attempt to do something was successful,
- chasing progress because they have not heard what is going on, a time limit has expired, or
- just generally worrying about something that they consider to be important.
An example from e-commerce is: calling to find out when a product will be delivered.
A “delayed impact” problem does not affect the result – yet
Some errors are not problems at the time of the transaction or activity that created data, but the data becomes problematic due to later events.
Let’s think for a moment about phone numbers. Have you changed a phone number for any reason? Perhaps, cancelling a landline. Or losing access to a work mobile because you changed jobs. Or maybe you lost a mobile, or had one stolen. Then think about the number of organisations who have asked you for a phone number and might now have an incorrect one for you.
Sometimes this doesn’t matter at all. For example, recently when I checked my credit card statement online, I decided to have a look at the personal data they held for me. I realised that one of the phone numbers was for a landline that I got rid of at least 15 years ago. It seems to have had no effect since then, which is good because there was no way for me to change it.
Sometimes it creates considerable hassle. For example, Lizzie Bruce documented the chats, phone calls, and paperwork she had to do after she unfortunately had to change her phone number – and had no access to the old phone.
But let’s not be distracted by these (relatively) minor gripes about the modern mobile phone. What about the documents, or lack of them, that affect people’s lives more profoundly because there has been no change in the data they have access to but the government has changed its mind about what is acceptable?
For example, the Windrush Lessons Learned Review is full of details where the data held or not held by individuals, and held or not held by government about those same people, was entirely acceptable and sufficient according to the law at the time of their arrival in the UK. But gradually the requirements changed as the law changed, leading to scandalously bad treatment of people, such as losing their jobs, their homes, or being forced out of the UK.
Errors can be caused by non-uptake or over-uptake
Non-uptake happens when a person who ought to do something doesn’t do it, such as failing to file a Self-Assessment tax return (which may result in a fine) or failing to claim a PAYE expense that they are entitled to (which doesn’t result in anything other than a loss to the person, but is still a policy failure).
Over-uptake happens when someone tries to do something twice, such as filing a return again because they don’t realise that the first one filed correctly, or trying to ‘cease’ something twice. Example from e-commerce: mistakenly buying the same thing again because of not being sure that the first attempt to purchase succeeded.