AI Product Psychology

Algorithm Aversion: One Mistake and the AI Gets Permanently Blocked

Last lesson was about calibrating trust. This one is about the biggest rock on that road: people’s tolerance for AI mistakes is far lower than for human ones. Same single error—a human colleague gets forgiven; the AI gets abandoned. The asymmetry has experiments, data, and an antidote.

Cast a voteAttribution translatorTweak-permission experimentAbandonment-rate dashboard
Hands-on · Next quarter, whose forecast do you trust?

You’re a sales VP with two sales-forecast sources. Last quarter they made exactly the same mistake: same numbers, same cause. Tap the one you’ll keep using next quarter.

Same mistake—which one do you forgive? Cast a vote
No right answer—go with your gut, then see if you match most people in the experiment
Human · Analyst Zhou
👨‍💼Zhou
Your sales analyst for three years
Last quarter he forecast East China sales at 50 million; actual was 44 million—12% too high.
In the retro he said: a competitor cut prices overnight—I genuinely didn’t see that coming.
AI · Forecast system
🤖SalesCast AI
Forecast system you’ve used for two quarters
Last quarter it forecast East China sales at 50 million; actual was 44 million—12% too high.
System log: competitor price-cut data was not included in this forecast run.
This bias has a name · Dietvorst’s experiment

Wharton, University of Pennsylvania, 2015. Dietvorst, Simmons, and Massey had participants forecast students’ academic performance—either themselves or via a statistical model—with a bonus for accuracy. The key twist: some participants first watched the model err. Those who saw the model err abandoned it in droves, preferring their own worse judgment—even when the data showed the model’s overall score beat humans by a clear margin. The paper’s title is Algorithm Aversion.

People don’t hate algorithms for being inaccurate—they hate them for having erred. Accuracy is a statistics problem; whether it has erred is a memory problem—and users decide with memory.
Hands-on · Attribution translator: is this about a person or AI?

The bias roots in how we attribute: for the same error, the ledger users keep for humans vs. AI differs. Below are six inner monologues—judge whether each evaluates a human colleague or AI. Finish all six and three psychological roots surface on their own.

Six monologues—judge who’s being evaluated 0 / 6
For each line tap “About a person” or “About AI,” then see the attribution pattern
Antidote experiment · A little control, and aversion fades fast

Dietvorst’s team followed up in 2018: same fallible model, but this time participants could tweak the model’s forecast—even by a little. Left: the interface they saw. Right: share who still chose the model. Switch the two settings yourself.

Tweak-permission experiment: switch two settings Flip it
forecast.example.com
Academic percentile forecast · Student #47
Model62
Your tweak62
In this setting the model result is final and cannot be changed.
Chose the model32%
Would rather do it themselves68%
Hands-on · Abandonment-rate dashboard: flip four levers

Bring the experiment back to your product. Scene: an AI expense-review assistant, one month live, just miscalculated a claim in front of a new cohort. The dashboard shows expected abandonment for that cohort; the four engineering levers on the right are switches you’ve already learned—flip them one by one, watch the needle, and read the principle inside each.

After the first error: how much abandonment can you still save? 0 / 4
Flip the switches on the right; the abandonment rate on the left moves live—open all four and see what’s left
Expected abandonment · this cohort58%
First error already happened; algorithm aversion at full force: over half plan to go back to Excel.
Current state: AI miscalculated a travel claim; the user stares at a read-only result page—can’t edit, can’t undo—and support says “the model does that sometimes.”
Quiz · A user just got burned by AI—what do you give them first?
A user just got burned by an AI mistake once—which move is most likely to win them back? Single choice
Wrong picks still explain; keep going until you hit the right one
APop an apology: “Sorry for the inconvenience—our model is continuously improving”
BCompensate: gift a month of membership to show sincerity
CLet them fix it: one-click correct the error; the system remembers the rule and shows “Saved—applies next time”
DShip a stronger model overnight and win trust back with accuracy
Don’t forget the other direction · automation bias

Algorithm aversion has a twin in reverse: automation bias. Another cohort accepts AI output unconditionally—even copying visible errors. Lesson 5 covered accidents from total trust. One product holds both types: averters need control and stability; blind trusters need friction and warnings. Trust calibration is bidirectional engineering—push only one end and the other blows up.

Key Takeaways
✓

Asymmetric forgiveness: human errors go on the situation ledger; AI errors go on the ability ledger. Same mistake—humans get forgiven, algorithms get abandoned, even when the algorithm’s overall score is better.

✓

The antidote is control: in the 2018 follow-up, merely allowing a tweak doubled model adoption—and most people barely moved the slider. What they wanted was the right to change it, not the change itself.

✓

Where the first error lands decides life or death: early-user mistakes trigger abandonment; late-user mistakes trigger grumbling. Put new users on the steadiest path; open experimental features only to veterans.

✓

Fall in the same hole only once: corrections users taught you must enter memory and show “Saved.” Crash in the same spot twice—aversion plus disappointment—and you’re basically unrecoverable.

Source: Original to Xiaoshan Academy's AI Product Psychology series; algorithm-aversion experiments from Dietvorst, Simmons & Massey, Algorithm Aversion (2015) and Overcoming Algorithm Aversion (2018).