Consumers are 32 per cent more likely to buy a product after reading an AI-generated summary of a review than after reading the original human-written text, according to a new study that quantifies how chatbots can subtly influence decision-making.
Computer scientists at the University of California San Diego found that large language models (LLMs) frequently introduce a “positive framing” bias when condensing information, stripping away nuance and making products sound better than they actually are.
In a paper presented at the International Joint Conference on Natural Language Processing, the researchers revealed that this subtle distortion has a measurable impact on human behaviour.
“We did not expect how big the impact of the summaries would be,” says Abeer Alessa, the paper’s first author. “Our tests were set in a low-stakes scenario. But in a high-stakes setting, the impact could be much more extreme.”
The polisher effect
The study found that in 26.5 per cent of cases, the AI summary effectively changed the sentiment of the original review, often by smoothing out complaints or amplifying praise.
This happens partly because models often rely heavily on the beginning of a text, ignoring crucial details or critiques that appear later.
To test the impact, the team recruited 70 participants to read either original reviews or AI-generated summaries of products such as headsets and radios. The results were stark: 84 per cent of those who read the AI version said they would buy the product, compared to just 52 per cent of those who read the raw human reviews.
Hallucination risks
Beyond shopping, the study highlighted significant reliability issues when chatbots are asked to handle facts outside their training data.
When answering questions about news items — both real and fake — that occurred after their training cutoff, the models hallucinated 60 per cent of the time, often confidently fabricating answers rather than admitting ignorance.
“This consistently low accuracy highlights a critical limitation: the persistent inability to reliably differentiate fact from fabrication,” the researchers write.
No easy fix
The team tested 18 different methods to mitigate these biases, but the results were mixed. While some fixes worked for specific models or scenarios, none were effective across the board, and some even made the models less reliable in other areas.
“There is a difference between fixing bias and hallucinations at large and fixing these issues in specific scenarios and applications,” says Julian McAuley, a professor of computer science at UC San Diego and the paper’s senior author.
