An internal assistant cites a legal article that does not exist. A customer chatbot announces a refund policy the company never had. A summarization tool attributes to a supplier a commitment that appears nowhere in the contract. In all three cases, the answer is well written, confident, and wrong.
This is called a hallucination: the model produces plausible but inaccurate information. It is the first obstacle to putting many AI projects into production, and rightly so. The good news is that we now know how to reduce them significantly, as long as we understand where they come from.
Why an LLM makes things up
A language model does not look up a database of facts. It generates text by predicting, word after word, the most likely continuation given what it learned and what it is given. When the information is present, in its training data or in the context, that prediction is usually right. When it is missing, the model does not stop: it produces the continuation that most “looks like” a good answer.
In other words, the model has no built-in way to tell “I know” from “I am guessing”. And since it was trained to be helpful and to answer, it often prefers a plausible answer to admitting it does not know.
A hallucination is not an occasional bug. It is what a model does when asked for information it does not have.
The situations that encourage them
- Questions about precise facts: dates, figures, legal references, names, quotes. The more specific the expected answer, the higher the risk.
- Recent or internal information: anything that was not in the model’s training data, such as your prices, procedures or products, must be provided to it.
- Incomplete or contradictory context: a RAG system that retrieves the wrong documents, or documents that contradict each other, pushes the model to fill the gaps.
- Instructions that force an answer: a prompt that demands “always answer”, or a rigid format with no “I don’t know” option.
- Long reasoning chains: a small error early on spreads and grows over the steps.
The levers that really reduce hallucinations
There is no magic setting. Reliability comes from a combination of design choices, each reducing part of the risk.
- Ground answers in your sources: this is the most powerful lever. With a well-built RAG system, the model answers from provided documents, not from memory. The quality of chunking and retrieval matters as much as the model itself.
- Require citations: ask the model to give the source of each statement, and automatically check that the source exists and actually contains the information.
- Explicitly allow “I don’t know”: a clear instruction, and a response format that provides for this case, clearly reduce invented answers.
- Narrow the scope: an assistant that only covers a defined domain, and politely declines the rest, hallucinates far less than a general-purpose one.
- Leave calculations and data to code: an amount, a date or a stock level should come from a reliable system (database, ERP, calculation), not be generated by the model.
- Structure the outputs: when the answer feeds another tool, a strict, validated format limits drift.
Measure before you deploy
You can only reduce what you measure. Before any go-live, we build with our clients a representative set of test questions, with expected answers and their sources. It deliberately includes trick questions: out-of-scope topics, information missing from the knowledge base, ambiguous phrasing.
This test set is used to compare settings, validate each change and track quality over time. Once in production, we add simple user feedback (helpful or not) and regular review of a sample of conversations. A hallucination rate is not declared: it is measured, and it must stay below a threshold agreed with the business.
A concrete example
A customer service team wants an assistant that answers questions about product warranty terms. The first prototype, built on a model alone, answers fluently… and invents warranty periods for products it does not know.
The deployed version changes three things: answers rely only on indexed product sheets and terms and conditions, each answer cites the document used, and the assistant is instructed to reply “I can’t find this information, I’m passing you to an advisor” when no source covers the question. The warranty periods themselves are read directly from the product database. On the test set, invented answers become rare, and those that remain are caught by citation checking.
Our recommendation
Do not ask “does this model hallucinate?”, because the answer is always yes to some extent. Ask instead: “is my system designed so that the model has the information, can say it does not know, and that its errors are caught?”. If you can answer yes to all three, hallucinations become a managed risk rather than a roadblock.