If you run a customer chatbot, you have probably heard that a fast handoff to a human is the safety net for the times it gets a fact wrong. This article looks at a new experiment that tested a version of that idea, what a scripted human apology and correction changed and what it did not measurably change, and what that suggests about where accuracy checks belong.
My view, which goes a little beyond the data, is that a missing human is not the main problem when a chatbot misleads someone. The wrong answer is.
Two experiments with a scripted shop chatbot
On October 27, 2026, Frontiers in Psychology published a peer-reviewed, open-access paper by Xi Chen, JiaRui Liu, QinXian Chen and Lin Li of Wuchang Shouyi University in Wuhan, China. They ran two scenario-based experiments about a shopping chatbot. Study 1 had 236 valid participants and Study 2 had 364.
The chatbot was not a live artificial intelligence (AI) system. The authors describe “a scripted chatbot with pre-written responses” and state that “No large language model (LLM) generated the experimental responses.” Participants viewed a shop page for a rice cooker, described as a top-selling product on the Taobao platform, and then asked the chatbot about it.
Each participant saw one of four versions. In the low-error version, the chatbot’s script contained two wrong statements. In the high-error version it contained six. One example in the high-error script was calling a 1.6 L cooker a “16 L” cooker. Both versions included a made-up offer of a free high-speed blender.
In half of the versions, a message then appeared that was presented as coming from a human customer service agent. It offered an apology for the error, corrected the facts, and advised the shopper to check the final checkout page. That message was pre-written too. No real person stepped in.
Two different reactions to being misled
The researchers asked participants to rate the service on 7-point scales. In Study 2 they also measured two reactions.
The first was technical frustration. The paper describes it as “a negative affective experience when users perceive that technology obstructs or delays progress toward a desired goal.” In a shop, the goal is a correct answer about the product, and a wrong answer gets in the way.
The second was cognitive dissonance. The idea comes from Leon Festinger’s 1957 theory, which the paper summarizes as “the psychological discomfort experienced by individuals when they hold two conflicting beliefs.” The authors argue that a chatbot’s errors break “users’ pre-existing presuppositions regarding the high intelligence of chatbots.”
Put simply, frustration is about the blocked task. Dissonance is about the belief that has to be revised. In my view, those are different problems, and there is little reason to expect one fix to reach both equally.
What the human message changed
More errors went with lower service ratings. In Study 1, without the human message, the average rating fell from 5.48 in the low-error version to 2.73 in the high-error version. With the human message, it fell from 5.77 to 3.87. The drop was smaller when the human message appeared, and that difference was statistically significant in both studies.
So the human message helped. Read the numbers the other way around, too. Even with the message, the high-error group’s average of 3.87 stayed well below both low-error groups. That comparison is mine, drawn from the reported averages; the authors did not test it as a separate result.
Study 2 shows where the help went. In the abstract, the authors write that the human recovery “effectively mitigates technical frustration but struggles to reverse cognitive dissonance.” In the statistics, the human message significantly weakened the link between errors and frustration, but the same test for dissonance was not significant.
The authors also modeled both reactions together. Both were linked with lower service ratings, and they describe these links as “indirect associations.” In their model, the two pathways together accounted for 66.8% of the total modeled effect of perceived errors on ratings. That figure is a share of a modeled effect, based on answers people gave about themselves. It is not a measure of how much of the harm is “psychological.”
What this study cannot tell us
The dissonance result is a non-significant result. It means the study did not detect an effect of the human message on dissonance. It does not prove there was no effect, and it does not show that the dissonance is permanent. The authors are careful here. They write that the study “does not yet allow us to conclude that human recovery is superior to AI recovery providing the same content, or that cognitive dissonance cannot be alleviated through other remediation methods.”
Most of the limits below come from the authors themselves.
- The chatbot was scripted, and the human agent was a pre-written message. The authors write that generalizability “to deployed LLM chatbots and real human handoffs remains to be established.”
- There was one product, a rice cooker.
- The task was short. The consultation was limited to three set tasks, and participants were told to finish within five minutes, so the results describe immediate reactions, not how service ratings develop over longer or repeated interactions.
- All measures were self-reported.
- There was no error-free comparison group. The test compared two errors with six.
- The paper names the Credamo platform as the recruitment source for its pilot test and Study 1, does not name the source for Study 2, and reports ethics approval from a university committee in China. It does not state where participants lived. In my view, the results may not carry over to other markets, products or kinds of service.
My interpretation
The authors’ own explanation is that cognitive dissonance “is highly stable and difficult to moderate through external compensatory services.” That is their reading, and the study did not test it directly.
My reading is simpler. An apology and a correction can fix the task. They cannot unsay the first answer. A shopper who was told a cooker holds 16 L, and then learns it does not, now knows the chatbot can be wrong about the product. If that holds outside a lab, a human handoff is good recovery, but it does not stand in for getting the answer right the first time.
The authors point in a similar direction. They recommend that platforms “strengthen source-level governance of AI hallucinations, particularly concerning core information that drives purchase decisions,” and say that strict knowledge base matching and review mechanisms are the benchmark for accurate and reliable fulfillment information. They also recommend clear handoffs from chatbot to human service, so the two ideas sit together.
What I would do with this
These are my suggestions, not the authors’ findings.
- List the facts that drive a purchase decision in your context, such as price, specifications, promotions and delivery terms. Check the chatbot’s answers on those facts against an approved source before the answers reach a customer.
- Keep the human handoff, and judge it on what this study suggests it does well, which is resolving the task and easing frustration.
- Do not treat a better satisfaction score after a handoff as proof that the customer’s confidence has recovered. In this study, ratings improved with the human message, while its effect on dissonance was not significant.
- As a shopper, check price, size and any offer on the final checkout page or the product page. Checking the final checkout page was also the advice in the study’s recovery message.
Pick the three facts your chatbot is asked about most often, and find out where its answers on them come from.
This article reflects the author’s personal views, not those of any employer, and is based on a publicly available peer-reviewed study by Chen, Liu, Chen and Li in Frontiers in Psychology, as available on October 31, 2026. It is not affiliated with or endorsed by the authors or Frontiers in Psychology, and quoted findings and opinions are theirs. It is general information, not legal or professional advice, and reading it does not create any advisory relationship. Get qualified advice before acting on how it applies to your organization.