On paper, RAG looks simple: index documents, retrieve the relevant passages for each question, hand them to the model, and it answers. A prototype takes a few days, and the demo is often impressive. Then come real user questions, badly organized documents, conflicting versions, and trust starts to crumble.

We have already explained when to choose RAG over fine-tuning. This article goes further: it describes, step by step, what separates a RAG prototype from a document assistant a team can genuinely rely on.

It all starts with the documents

The quality of a RAG system is capped by the quality of its knowledge base. Before indexing anything, we take inventory with business teams: which sources are authoritative, which are outdated, who owns them.

  • Remove outdated versions: an old procedure indexed next to the new one guarantees contradictory answers.
  • Name a reference source when several documents cover the same topic.
  • Convert cleanly: a scanned PDF, a badly extracted table or a header repeated on every page degrade retrieval. Text extraction deserves as much care as everything else.

Chunk along the structure

A model does not receive whole documents, but chunks. How those chunks are cut directly affects answer quality. The most common approach, fixed-size blocks, is also the riskiest: it splits a rule in two, separates a table from its title, isolates an exception from the rule it applies to.

We chunk along the document’s structure: headings, sections, clauses, questions and answers. Each chunk keeps its section heading and document name, so it stays understandable on its own. For very long content, a slight overlap between chunks avoids losing context at the boundaries. There is no universal ideal size: it is tuned to the document type and validated through testing.

Metadata and access rights

Each indexed chunk carries useful information: source document, date, version, owning team, content type, language. This metadata is used to filter retrieval (only documents in force, only the scope of one country or product line) and to display clear citations.

Above all, it is used to enforce access rights. An internal assistant must never become a shortcut to documents a user is not allowed to see. Permission filtering must happen at retrieval time, before a single chunk is sent to the model, not by asking the model to “not mention” certain information.

A RAG system should never know more than the person asking it the question.

Hybrid search, then reranking

So-called semantic search, using vectors, finds passages close in meaning, even when the wording differs. It is valuable, but it has a weak spot: exact terms. A contract number, a product reference, an internal acronym or a proper name are often better found by classic keyword search.

So we combine both: keyword search and vector search, with their results merged. Then comes a reranking step, which re-scores the best candidates against the precise question and keeps only the most relevant. Sending the model five genuinely useful passages is far better than sending twenty, half of which are noise.

Verifiable citations

Every answer must show the documents it relies on, with a link to open them. That is what lets users check, and therefore trust. We also automatically verify that the cited sources are indeed among the passages given to the model, to catch an invented citation.

Knowing how to say “I didn’t find it”

A reliable assistant is one that admits its limits. When retrieval returns nothing relevant enough, the right answer is not an approximate one, but a clear message: the information was not found in the knowledge base, with a suggestion for next steps (rephrase, contact the relevant team). This behavior is configured: a relevance threshold below which the assistant does not answer, an explicit instruction, a response format that provides for this case.

These unanswered questions are also a goldmine: they reveal gaps in the documentation, to be filled first.

A living index

Documentation changes. A RAG system that is not synchronized with its sources ends up answering with outdated information, which is worse than not answering. The index must follow the document lifecycle: adding a new document, updating a version, and above all deleting. When a procedure is withdrawn, all its chunks must disappear from the index. This is also a GDPR requirement when personal data is involved.

In practice, we set up automatic synchronization with the source document spaces, and track what was indexed, when, and from which version.

An evaluation set from day one

Before tuning anything, we build with the teams a set of representative questions, each with the expected answer and the document that contains it. We add out-of-scope questions and questions whose answer does not exist in the base. This set measures two things separately: did retrieval find the right passage? And did the model draw the right answer from it? Without that distinction, you do not know what to fix.

A concrete example

A services company wants an assistant for its support teams, based on several hundred procedure sheets and contract terms. The first prototype, built quickly, gives good answers in the demo, but regularly confuses offers with similar names and sometimes cites an old set of terms.

The deployed version revisits every step: old versions are removed from the base, sheets are chunked by section with their heading, each chunk carries the offer it concerns and its validity date. Retrieval combines keywords and vectors, which resolves the confusion between offer references, and reranking keeps only the best passages. Every answer cites its source sheet, and the assistant points to a referent when it finds nothing. The evaluation set, re-run after every change, checks that fixes do not break anything else.

Our recommendation

The language model is only one part of a RAG system, and rarely the one causing problems. Invest first in the knowledge base, chunking, access rights and evaluation. It is less spectacular than switching models, but that is where reliability is won, and with it, real adoption by your teams.