How to catch AI when it makes things up

Everyone who works with AI has done it at least once. You ask a question, the answer comes back clean and well organised, you paste it into a document, and a day later someone points out that one of the facts in it was invented. The model was not lying. It has no concept of true, and none of false. It produces the words most likely to come next, and likely words go wrong more often than they look.

You will not train that habit out of the tool, so the job is to catch it yourself. In the teams we work with, the people who get dependable work out of AI are rarely the ones with the best tools. They are the ones running a method while everyone else improvises. Here is the one we set up first. Four habits, and one honest limit.

Why a wrong answer sounds so right

Underneath, a language model does not look anything up. Tools that bolt on web search or document retrieval change what it can reach, but the model itself still works the same way. It generates text one token at a time, each one the most likely continuation of what came before, and nothing in that process checks whether the result is true. When the patterns match reality you get a correct answer. The rest of the time you get a fabrication, delivered in the same calm voice.

There is usually no reliable tell in the writing. A confident sentence about a real regulation and a confident sentence about a regulation that does not exist often look the same on the page. So “read it carefully” fails as a strategy on its own. You need a procedure that works even when the error is invisible.

Be the expert in the room

The first habit costs nothing. Point AI at work you can already judge. Ethan Mollick puts it plainly in Using AI Right Now: use AI for topics you understand until you have a feel for its capabilities and its failure modes. When you know the territory, you notice the invented street name. When you do not, you have handed your judgement to a system that has none.

It also changes how you work with it. The people who get the most out of AI treat it as a back and forth. They push, they challenge the first answer, they ask for the reasoning behind it. Most people do the opposite. One question, no context, default settings, and they take whatever comes back first. The reliable move is to interrogate the draft rather than quote it.

Ground every claim

Expertise tells you when to be suspicious. Grounding gives the model less room to invent in the first place. The most reliable thing you can do is stop it answering from memory and make it answer from sources you hand it. A review of fact-checking methods across the research literature (Rahman et al., 2025) reaches the same conclusion: grounding outputs in external, verifiable sources is the leading way to cut factual error. Anthropic’s own guidance on reducing hallucinations turns that into four moves you can run by hand.

Let it say “I don’t know”

Give the model explicit permission to admit uncertainty. It does more than it sounds. A model under implicit pressure to produce an answer will produce one whether or not it has the basis for it. Tell it that “I don’t have enough information to answer this” is an acceptable response, and much of the confident nonsense stops appearing.

Quote first, answer second

On a long document, ask for the exact passages first, word for word, and have the model answer only from those quotes. Pulling the evidence before the conclusion anchors the response in the actual text instead of a paraphrase the model half-remembers.

Cite every claim, retract the rest

This habit does the most work. Once the draft exists, have the model find a supporting quote for every claim it made, then drop any claim it cannot support. Anthropic’s worked example marks each removed claim with empty brackets, so you can see where the draft was running on confidence instead of evidence.

Restrict it to your sources

Tell the model to use only the documents you supplied and to leave its general knowledge out of it. That keeps a hard line between what the brief actually says and what the model assumes it knows, which is the line fabrications cross.

Triangulate, don’t take the first answer

Output from a probabilistic system is not stable, and you can use that. Run the same prompt a few times and compare. Where the answers agree, you have a reasonable signal, but not proof. A model can repeat the same fabrication on every run, because that wrong answer was the most likely one to begin with. Where the answers diverge, you have found a soft spot worth checking by hand.

Research backs the instinct, in a narrower setting than you might assume. When a question has a single checkable answer, sampling several independent reasoning paths and keeping the majority result measurably improves accuracy. In one controlled study that approach raised performance on a grade-school maths benchmark by 17.9 points (Wang et al., 2023). That worked because each answer was a single checkable number, so comparing prose drafts is a looser version of the same idea rather than the same thing. Treat agreement between runs as support for the habit, not a guarantee, and treat disagreement as a flag worth chasing.

You can also ask the model to critique its own draft against the sources, and it will catch some of its own mistakes. Do not give it the last word. A model grading its own work tends to go easy on it, so the final check should come from outside the model, whether a fresh session, the source document, or another person.

Edit the prompt, not the output

When you catch a bad output, the tempting move is to fix that one artefact by hand. Resist it. The output is disposable. The prompt is what you re-run, so the prompt is where a fix actually pays off. Correct the failure at its source and every later run inherits the fix.

Simon Willison draws a sharp line in Hallucinations in code are the least dangerous form of LLM mistakes. With code you get fact-checking for free. Run it, and a hallucinated method throws an error on the spot. Prose has no such safety net. As he puts it, for regular prose “you need a critical eye, strong intuitions and well developed fact checking skills to avoid sharing information that’s incorrect and directly harmful to your reputation.” The further your work sits from anything a machine can check for you, the more it rests on your own method.

The one limit, stated plainly

That caveat is the honest part, and it is why none of this works as a one-time trick. It only pays off as a standing habit, run after run. Grounding, citing, comparing, and fixing the prompt instead of the output are small disciplines on their own. Practised every time, they are what make AI dependable enough to build real work on.

This pays off whether you work on your own or across a team. On your own, it is what separates output you hope is right from output you have actually checked. Across a team, the same habit written down and shared stops being a knack a few careful people have and becomes something everyone applies the same way. That is where the bigger risk gets handled, the work that would otherwise ship without anyone checking it.

Setting up that standard, and the judgement to know which work needs which level of checking, is what we do alongside the people and teams we work with. If that is the gap you are closing, see how we work with teams or with you.

Sources

• Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023. arxiv.org/abs/2203.11171
• Rahman, S. S., Islam, M. A., et al. (2025). Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Models. Artificial Intelligence Review (Springer). arxiv.org/abs/2508.03860
• Mollick, E. (2025, June 23). Using AI Right Now: A Quick Guide. One Useful Thing. oneusefulthing.org
• Anthropic. Reduce hallucinations. Claude documentation. platform.claude.com
• Willison, S. (2025, March 2). Hallucinations in code are the least dangerous form of LLM mistakes. simonwillison.net