Why ChatGPT makes things up, and how to catch it

ChatGPT will sometimes state something false with complete confidence. It may invent a statistic, a court case, a quotation or a book that does not exist. This is usually called a hallucination. It is not a bug that will soon be fixed; it follows from how these tools work.

Why it happens

A language model generates text by predicting what is most likely to come next, based on patterns in the enormous amount of writing it was trained on. It does not consult a database of facts. When the answer to a question was common in its training data, the most likely continuation is usually correct. When the answer was rare, recent or never written down, the model still produces the most plausible-sounding continuation. Plausible and true are not the same thing.

The model has no reliable internal signal that tells it when it is guessing. That is why a hallucination reads exactly like a correct answer. Tone is not evidence.

The answers most likely to be wrong

• Specific numbers: statistics, percentages, prices, dates.
• Sources: titles of papers, books, articles and legal cases, and the page or section numbers attached to them.
• Quotations attributed to a named person.
• Recent events, and anything that changed after the model was trained.
• Niche facts about small organisations, local rules or specialised fields.
• Summaries of a document the model was never actually given.

What reduces the problem

Give the model the source material instead of asking it to remember. "Summarise the attached report" is far safer than "What does the 2025 report say?" Tools with web search can cite pages, which helps, but the citations still need opening: models sometimes attach a real link to a claim the page does not make. Asking the model to say "I do not know" when it is unsure helps a little. It does not solve the problem.

A simple checking routine

• Decide what an error would cost. A wrong word in a birthday message costs nothing; a wrong figure in a client report costs a great deal.
• For anything costly, identify every specific claim: each number, name, date, quotation and source.
• Check each one against an original source that you open yourself, not against another AI answer.
• If a source cannot be found, treat the claim as false until proven otherwise.

This sounds slow. In practice it takes a few minutes, and it is far faster than repairing the damage from a confident error that reached a client or a manager.

Where AI is still useful

Hallucinations matter least where you can judge the output quickly: drafting, rewording, summarising material you supply, brainstorming and structuring your own ideas. They matter most where the value of the answer lies in facts you cannot easily verify. Choose tasks accordingly.

Module 5 of the free Course Parse course covers hallucinations in more depth, with practice examples, and it is available once the first module is complete.