Why AI Chatbots Would Rather Make Things Up Than Say 'I Don't Know'

Ask an AI chatbot a factual question it cannot answer and it rarely replies “I don’t know.” More often it produces a confident, fluent, and completely invented response. Researchers call this hallucination, and a new paper posted to arXiv, the preprint server where scientists share early findings, offers an information-theoretic explanation for why it happens.

The core argument is surprisingly intuitive. When a language model is trained to produce likely-sounding text, guessing something plausible is often a safer bet than refusing. A flat “I don’t know” carries little information and, in the training data, appears far less often than a fluent answer. The paper frames the problem in terms of the space of possible outputs: models are optimized to minimize error on the text they see, and that objective quietly rewards confident-sounding fabrication over honest silence.

A detailed discussion of the work appeared on Zhihu, a Chinese question-and-answer platform often described as the country’s answer to Quora, where one of the paper’s authors walked readers through the motivation behind the research. The author described how the project began with a critical reading of earlier work on model behavior, including analyses of why language models hallucinate, and gradually moved from abstract bounds in information theory toward the practical puzzle of chatbots that will not admit ignorance.

The explanation matters because hallucination is one of the biggest obstacles to trusting AI systems. A model that invents a legal citation, a medical dosage, or a historical date can mislead users who have no easy way to check. If the tendency is baked into how these systems are trained and scored, then simply telling users to “verify everything” is not a complete fix.

One implication is that the fix may need to start at the training stage. If models were rewarded for expressing uncertainty, or if refusing to answer were treated as a valid and even desirable outcome, the incentive to fabricate would weaken. Some researchers have proposed teaching models to say “I’m not sure” more often, though doing so without making them unhelpfully cautious is a delicate balance.

The Zhihu thread also touched on a broader theme: how technical research travels into public understanding. A paper written for specialists can reach millions of readers when an author or a knowledgeable commenter translates it into plain language. In this case, the discussion attracted attention not because of any product launch but because the question — why does my AI make things up? — is one that ordinary users ask every day.

For now, the paper is a preprint, meaning it has not yet completed formal peer review, and its conclusions will be tested and debated by other researchers. But the framing is useful: hallucination may not be a bug that can simply be patched out. It may be a predictable consequence of how these systems are built and scored, and understanding that is the first step toward building AI that knows the limits of what it knows.