A call for the resignation of the customer satisfaction survey
The score everyone celebrates tells you nothing. Here's how to fix that.
A near-perfect customer satisfaction score should feel like a win. Leadership loves it, annual reviews beg for it … so why am I going out of my way to gather worse scores? Let’s break down why customer satisfaction scores (or CSATs) need to go in the bin, and how you can truly discover what your customers are experiencing.
I was preparing dinner for some friends last month. My partner’s best friend was coming over, and so was a last-minute addition: the new boyfriend, someone neither of us had met. So, what to cook?
Does he eat meat? “I don’t know.” Vegetarian? Any allergies? “Pfff, no idea.” Does he like a bit of spice? “Pass …” What my partner can tell me, in surprising detail, is that he’s deep into science fiction (the Alien franchise especially), yet he’s confusingly also a devout flat-earther (this should be an interesting evening …), and he fanatically follows a football team whose name “I think begins with an E, or something” (the detail dropped off on that last point). Not exactly useful information in this context.
Wait … I’ve got it – pizzas! Everyone likes pizza (especially flat-earthers?) and if I make a few little ones I can improvise and adapt to tastes, swapping ingredients in and out. Foolproof! Andy, you’re a genius!
Pizza 1: margherita. Classic, not adventurous, a safe baseline. The new guy’s response … “Lovely!”
Pizza 2: prosciutto e funghi. Still classic, slightly more risk. The review … “Lovely!”
Pizza 3: anchovies. Divisive, dangerous, but my personal favourite, so I’ll happily eat it all if the group isn’t keen. I get one slice, because … “Lovely!”
And on it went. The pizza plan paid off, and everything was indeed “lovely”. Fast forward to our next gathering and I’m staring into the fridge again wondering what to cook. I know he likes pizza, but I can’t repeat myself – that’d be boring. Italian, then? That’s an assumption, not a certainty. And which pizza did he actually prefer? No idea. He just told me they were all lovely.
This is exactly the level of information that Likert-based customer satisfaction scores provide (you know, those strongly-agree-to-strongly-disagree ladders). They hand you a marker of ‘loveliness’ with no depth beneath it, so the next time you sit down to cook, you have no concrete information to work with.
Customer satisfaction scores: the good, the bad, the ugly, and the misleading
Whenever you’ve completed some form of professional education, you’ve very likely been handed a survey about your learning experience. The course was useful. I understood the material. The length was about right. You tap “agree”, maybe “strongly agree” and you close the tab.
For a year, a learning program I joined was defaulting to exactly this type of survey. At the end of the year we mapped strongly disagree to 1 and strongly agree to 5 and took the averages. We hit 4.5 out of 5 for learner satisfaction across thousands of responses, comfortably over the 4.2 target. Leadership loved it, we loved it, it seemed like such a triumph … at surface level.
On a deeper level, the surveys were bloody useless. The average for every question, on every course, sat somewhere between 4.3 and 4.7. When nearly everyone agrees with nearly everything, you haven’t measured the experience; you’ve measured people’s reluctance to be unkind on their way out. I knew the learners were content (and, victoriously, much more content than last year) but I had no specific insight into why, what exactly had I done to drive this change, or, crucially, what can I change to make things even better and more effective in the next design. In short, I had no ability to meaningfully distinguish between the customer’s experience of all the learning products I’d made throughout the year because they were all … lovely.
Learning designers lean on satisfaction scores because we’re desperate to quantify our impact, and they’re the simplest, most accepted way to do it. But when you deliver something holistic and human, and then try to squeeze it into a number to prove it worked, you often show nothing at all – and instead lead yourself into the dangerous trap of believing your own positive publicity.
The good news is that you can have both the numbers and the information you need to drive improvements. You just need to structure the right questions.
Poor questions give poor answers
Two things were sabotaging my survey design.
The questions were leading. For example, “How much do you agree with the following statement: the e-learning was useful and improved my understanding.” That’s the survey equivalent of leaning across the table and asking, “Hey! New guy! The pizza was delicious, right?!” Nobody’s going to say no. It isn’t a question, it’s a nudge in a lanyard, and agreeing is the path of least resistance.
The scale was mush. Like most satisfaction surveys, we used a Likert scale – the strongly-agree-to-strongly-disagree ladder that asks you to mash a real, textured opinion into one of five options. Then I’d turn the agreement statement into a number, average all the numbers, and report one figure. Meaning leaks out at every step, until “4.5/5” has almost none of the experience left inside it.
I got so frustrated with how little my own surveys were telling me that I set out to build a solution from scratch – something that would give me the information I needed, while avoiding stakeholders asking “Where did the CSATs go? We liked those. We need numbers.”
Thankfully I’m not the only one out there who is dissatisfied with customer satisfaction surveys. Will Thalheimer is an expert in how we evaluate learning, and he makes a convincing case against Likert-based satisfaction surveys. The research he pulls together is genuinely damning: conventional satisfaction scores barely correlate with how much people actually learn or how they later perform (Alliger et al., 1997; Sitzmann et al., 2008). Agreement scales, he argues, are ambiguous, tiring, and tend to produce the same weary tap straight down the page – agree, agree, strongly agree, agree … done.
It was time to stop chasing a better score and start designing for insightful answers. Less “was the meal lovely?”, more “how was the amount of salt for you?”.
Good questions give good answers
The redesign – which I’ve coined Actionable Experience Surveys – rebuilt survey design from the ground up based on three key principles.
Bin the leading statement. Ask something neutral and open. Not “the practice questions were useful (agree?)” but “which of these statements best describes how the practice questions worked for your learning experience?” No thumb on the scale.
Swap the agreement ladder for real sentences. Instead of five shades of “agree”, the learner picks one statement that concretely says something – from “the practice questions were too complicated” to “the questions were useful, but I need more of them”. Every option is there to direct you to an action, e.g., reduce complexity, or add more questions.
Tag each sentence to a standard. Behind the scenes, every statement that the customer can select is marked as Alarming, Concerning, Acceptable, or Superior. This is the move that lets you keep the numbers and the meaning all at once:
The quantitative half: I could still report “72% at or above the acceptable/superior bar”, track it over time, and set targets – the numerical benchmarking leadership rightly wants.
The qualitative half: because the options are specific sentences, the spread itself is a diagnosis. You don’t just learn how many were unhappy. You learn exactly what the issues are.
Why it works: two quiet bits of behavioural science
Here’s where the design earns its keep.
It forces metacognition – thinking about your own thinking. A Likert ladder lets you freewheel; in fact, it invites you to skip evaluation altogether. In AESurveys, four or more distinct, plausible sentences make you stop and genuinely weigh up what you got from the thing. That small cognitive friction isn’t a bug; it’s the entire point. It pulls a considered judgement out of people instead of a reflex.
It primes the free text response. Every question ends with an optional comment box – nothing new there, we did that with the satisfaction surveys too. But what comes before the box decides what goes in it. This is priming: earlier input shaping later response. After a vague “agree”, you get a vague comment (if you get one at all). After choosing between four or more sharp, specific sentences, the reader’s mind is already running in the right channel, so the comment comes out focused and on topic. I didn’t get richer free text by begging for it. I got it by making people think properly about the question first, so they’re ready and willing to give their opinion which has formed naturally.
No design is free though. Making people think harder is more work, and more work means some will bail before finishing. I manage that by keeping surveys short and scaled to the course – maximum five minutes of survey for an hour of learning. I haven’t seen any drop off in participation, but you’ll have to judge this for your use case.
Actionable Experience Surveys: the good, the bad, the ugly, and the informative
Rebuilt this way, my surveys stopped flattering me and started feeding me. Same people, same broad goodwill – but now I could truly understand their experience and act to improve it (hence “Actionable Experience Surveys”, or AESurveys). For example, I wanted to dive deep on how the examples embedded in a learning program were serving my learners. The customer satisfaction survey would have told me they were ‘lovely.’ The AESurvey showed me that a strong majority (64%) believed the examples had directly improved their work processes (my ‘Superior’ AESurvey marker). The remaining 36% didn’t simply contradict the “superior” responses, instead they pointed to specific fixes. For example, 12% were asking for more examples, 10% were asking for greater complexity in the examples. So, alongside the pat on the back from the 64%, I received a prioritised to-do list which I can act on to push the 64% higher.
What you can take away
When you build AESurveys from scratch you can use the answer statements to truly zoom in on what you need feedback about (it’s particularly effective to use learning objectives to structure AESurvey questions for learning programs). And if you already have surveys, you can also rebuild almost any question into the AESurvey format. For example, let’s imagine I’ve gone pro with my pizza, and I’m now running a focus group for my pizza restaurant. My customer satisfaction survey could look like this…
“The pizza crust has the perfect balance of crispiness and chewiness.”
Strongly disagree
Disagree
Neutral
Agree
Strongly agree
… but that would have put me back in the ‘lovely’ void. If I’m not hitting 100% strongly agree, I’m left with an ambiguous problem to deal with – too crispy, or too chewy?
Let’s make it an AESurvey question by avoiding bias — “What did you make of the pizza crust?” — and structuring the responses so I can read crispiness against chewiness. Each response maps to a category (Alarming, Concerning, Acceptable, or Superior), the respondent only sees and selects the statement:
Negative
Alarming — “I’ve nothing positive to say about the crust” · 5%
Concerning — “Too chewy” · 5%
Concerning — “Too crispy” · 20%
Positive
Acceptable — “Good, but I’d prefer it crispier” · 15%
Acceptable — “Good overall, but I’d prefer it chewier” · 25%
Superior — “Nicely balanced between crispy and chewy” · 30%
Note how I’ve aligned the information I need (the statements the respondents choose) against the AESurvey’s four categories — this is so I can give the appropriate audiences the right levels of information they need. I can tell my investors that 70% of customers are happy with the pizza crust (all the positive answers in my ‘Acceptable’ and ‘Superior’ categories). But I can go into the details and tell myself that there’s an issue with crispiness. Even in the ‘Acceptable’ range, it’s biased towards being too crispy. And in the ‘Concerning’ range, it’s heavily biased towards excessive crispiness. What’s the learning? Reduce crispiness to improve the customer experience.
I get a spectrum of detail within one question that I can present in full detail or in highlights according to my stakeholder requirements. By the time I’ve asked about the sauce, the toppings, the wine pairing and so on I will have a rich tapestry of data-driven action points, or I can reduce it to “65% of customers are responding positively to the restaurant and we have a plan in place to reach 75% by Q4.”
A quick recipe for an AESurvey question:
Delete the leading statement. If your question signals the “right” answer, rewrite it as a neutral one. Bias isn’t welcome.
Write real sentences, not a scale. Four or more options a real person might actually say. If picking one doesn’t give you the information you need, it’s too vague. When you’re reviewing the answers, you want to discover at least one clear action item.
Tag each response option to a standard. Alarming / Concerning / Acceptable / Superior. Two negative, two positive. Now you can count and diagnose, delivering the headline metrics to the business folks.
Add free-text boxes. The questions prime deeper thought, and those specifics are goldmines. (If you’re fielding thousands of responses, use AI to spot trends and summarise.)
No matter your industry, there’s always an ask from a higher power to reduce a rich, human experience to a number (whether that’s consuming education or pizza). It’s easy to get lost in the qualitative-versus-quantitative divide and believe you can only have one or the other. That’s a false choice. Design the questions well enough, and the number stops standing in for the meaning and starts carrying it. It is entirely reasonable (sensible, actually) to ask your learners directly, and specifically, for the insight you need. That’s not an imposition, it’s the whole opportunity. And if you close the loop, showing your customers how their input is improving their own experience, you start to build a genuinely healthy improvement ecosystem.
Beyond learning experience
The Actionable Experience Survey method isn’t tied to learning design evaluation. It’s about the gap between measuring a vague feeling and gathering specific, actionable insight – and that gap is everywhere the humble agreement scale has crept in:
Customer feedback
Product and UX research
Employee engagement and pulse surveys
Patient-reported experience, event feedback, 360 reviews
A caveat, because no method is a cure-all: AESurveys earn their keep when you need actionable depth. If you just want a demographic fact or a quick temperature check, a plain question still wins. Reach for AESurveys when the answer has to tell you what to do next, not merely how people felt.
And that’s that – next time you’re building a survey, or hunting for a clean read on how your content is being experienced, consider AESurveys. Done well, it’s a method of investigation that turns a polite shrug into a plan – and keeps the richness of meaning in, even while it’s handing you a number. Customer satisfaction survey, it’s time to retire.
A quick note on tooling: I originally built these as self-contained, portable survey modules so they weren’t welded to one platform. I happened to use Articulate Storyline, but nothing here depends on it – Adobe Captivate, Lectora, or a well-built form in Qualtrics, Typeform, or Google Forms will do the same job. The design is the asset, not the software.
Sources and further reading
Thalheimer, W. (2016). Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form. Somerville, MA: Work-Learning Press. (Later updated and retitled as Performance-Focused Learner Surveys, 2022.)
Thalheimer, W. (2018). The Learning-Transfer Evaluation Model (LTEM). Work-Learning Press. Available at worklearning.com.
Alliger, G. M., Tannenbaum, S. I., Bennett, W., Jr., Traver, H., & Shotland, A. (1997). A meta-analysis of the relations among training criteria. Personnel Psychology, 50(2), 341–358.
Sitzmann, T., Brown, K. G., Casper, W. J., Ely, K., & Zimmerman, R. D. (2008). A review and meta-analysis of the nomological network of trainee reactions. Journal of Applied Psychology, 93(2), 280–295.
Note: the meta-analyses above find the correlation between end-of-course reaction scores and actual learning or job performance to be very weak rather than literally zero; the exact figure varies by study.