Study finds unreliable AI advice made users less accurate, less likely to admit uncertainty, and more confident

Researchers from French and Italian universities found that access to AI advice cut participants’ “I don’t know” responses from 44% to 3%, while accuracy fell from 27% to 9% and confidence rose from 30% to 76%. The experiment used questions where the model was usually wrong, showing that participants often accepted bad answers even when they might have answered correctly on their own. Monetary incentives improved results only modestly, leaving accuracy and uncertainty recognition below no-AI baselines.

The findings highlight a risk of cognitive surrender: AI tools can erode users’ habit of pausing, checking uncertainty, or declining to answer. As AI assistants spread into education, work, and search, product design may need to emphasize calibrated confidence, source checking, and friction around low-reliability answers rather than seamless completion.