"The AI understood my question." We hear some version of this in almost every first demo of a LLM. It's not an unreasonable thing to say: the answer was relevant, well-written, on topic. Everything about the output looks like understanding.
The problem is that the phrase leads to bad decisions: trusting an answer because it "sounds" understood, writing prompts assuming the model guesses an unstated intent, being surprised when it "makes up" a fact with the same confidence as a correct one. Understanding what actually happens changes how you should use these tools, and how much you should trust them.
What a LLM actually does: predict the next word
A large language model doesn't "read" a sentence the way we do. It breaks it into tokens, computes a probability for every token that could come next, picks one, appends it to the text, and repeats the process, token after token, until the answer is complete. That's it. There is no step in this cycle where the model builds a verified, true representation of the question — there is a statistical calculation, trained at massive scale to produce plausible continuations.
Why it still feels like understanding
A model trained on hundreds of billions of words captures, in its probabilities, an enormous amount of the regularities of human language — which themselves reflect an enormous amount of reasoning, facts and logical structure already written down by humans. Predicting the statistically likely continuation after a well-known reasoning pattern reproduces that reasoning, because it exists countless times in the training data. The result looks like understanding because it faithfully reproduces its written trace, without the underlying mechanism being understanding at all.
A LLM doesn't know what it just said. It only knows that the continuation it produced was, statistically, a plausible one given what came before.
What this changes in practice
Nothing in a LLM's architecture distinguishes a plausible-but-false continuation from a plausible-and-true one — which is exactly what produces hallucination. A vague prompt leaves the model statistically free to continue in any plausible direction; explicit context, format and examples narrow that space and make outputs far more consistent. It's also why RAG works: injecting the right document into the context makes the correct answer statistically more likely than any other, because it is literally present in the text the model just read. And it's why any answer used for a decision that matters should be checkable against a source, never accepted just because it sounds confident — confidence is a byproduct of learned writing style, not a signal of truth.
In summary
A LLM doesn't understand your question — it predicts, token by token, the most statistically plausible continuation given everything it has read. That mechanism is powerful enough to look like understanding most of the time. Knowing it isn't changes everything: how you write a prompt, why RAG works, and why no answer deserves blind trust just because it's well phrased.
