Evaluating a RAG System Before You Let Anyone Use It
Every retrieval augmented generation project hits the same moment. The thing answers questions, the demo lands, and someone asks whether it can go live. That question deserves evidence, not impressions. Here is how we evaluate a document-grounded assistant before anyone outside the build team starts depending on it.
Why a Demo That Works Tells You Almost Nothing
The first five questions you try are ones you already know the answer to. That is exactly why they pass. Retrieval breaks on the long tail: the acronym one department uses and nobody else does, last quarter's policy revision, the document living in three conflicting versions. Two failure surfaces belong to you, and you need to keep them apart - retrieval pulled the wrong passages, or generation misused the right ones. Decide beforehand what the assistant has to get right, and where a refusal is a perfectly fine answer.
Build the Evaluation Set From Real Questions
Pull questions from support tickets, internal chat, onboarding material and the people who will actually use the thing. Never from the documents themselves. A question written while reading a document only tests whether you can find that document again, which is a much easier job than what users throw at you. Record the expected answer plus the source document and section, so retrieval gets scored separately from wording.
Include the hard categories on purpose:
- Questions with no answer anywhere in the corpus
- Questions with several equally valid answers
- Questions spanning two documents that must be combined
- Questions where the right response is to name a person
What to Actually Measure
- Retrieval quality: is the correct passage present at all, and how far down the ranking?
- Groundedness: every claim traceable to a retrieved passage, verified by reading the source rather than trusting that a citation appeared.
- Usefulness: complete enough to act on, scoped properly, not a hedge that restates the question back at you.
- Refusal behaviour: confident invention where the corpus says nothing destroys trust faster than anything else on this list.
- Latency and cost per question under realistic concurrency, not on an idle machine.
Reading the Citations, Not Just Counting Them
A source attached to an answer is a testable claim. So test it. Open the passage and confirm it says what the answer says it says. Go hunting for the common pattern: the retrieved chunk is topically related, but the specific number, date or condition came out of the model's general knowledge. And chunk boundaries produce their own bug class. A passage ending mid-clause can flip the meaning it looked like it carried.
Tip: have a domain expert, not an engineer, read twenty answers alongside their sources. They catch the plausible-but-wrong ones an engineer reads straight past.
Testing the Corpus, Not Only the Model
Most retrieval failures we trace back turn out to be document problems. Superseded versions still sitting in the index, scanned pages with no text layer, tables flattened into unreadable rows. Check what fraction of the corpus is retrievable at all - documents that never surface are dead weight dressed up as coverage. Permissions belong in the test plan too. Run identical questions as users with different access and confirm the answers diverge where they should. And settle ownership before launch: who removes an outdated document, and how fast does that reach the index?
Deployment Shape Changes What You Have to Verify
Cloud and on-premise installs share evaluation questions but not risk checklists. For self-hosted work, verify nothing leaves the network - embedding calls, telemetry and error reporting included. Local hardware caps model size and concurrency, so measure quality on the model you will actually run. Not the biggest one you trialled.
Tip: rerun the evaluation set after any infrastructure move. Swap an embedding model or a chunker and retrieval shifts quietly, even when nobody touched application code.
From Pilot to Production Without Guessing
Start with a small group who know the material well enough to spot a wrong answer, and give them one-click flagging. Log every question, its retrieved passages and the answer, so a complaint turns into a reproducible case instead of a description of one. Flagged answers become a growing regression set that every later change runs against. Agree on rollback triggers up front, and keep the assistant visibly labelled as assistive.
Frequently Asked Questions
How many test questions do we need before launch?
Enough to cover the question types users will genuinely ask, with real coverage of the hard categories. A small set drawn from actual tickets beats a large set generated from your own documents.
Can we use a language model to grade the answers?
It scales well once you have calibrated it against human judgements on a sample. As the only signal, it stays unreliable for anything where being wrong carries a cost.
How often should the evaluation run again?
On every change to the corpus, the chunking, the embedding model, the retrieval settings or the generation model. Each of those shifts results on its own.
What Good Enough Looks Like
The evaluation supports one decision: does this assistant help more than it misleads, for this corpus and these users? It is not a gate you clear once. It is the instrument you keep pointed at a system whose inputs keep changing. Done properly, you end up with a team that knows exactly where the assistant is weak and says so out loud - and that is what earns trust in the parts that work.
