Table of contents
Most teams building RAG judge it by trying a handful of questions and deciding the answers look good. That works for a demo and fails the moment you change a chunk size, swap an embedding model or add new documents, because nobody can tell whether the change helped. Evaluation is what turns RAG from a collection of guesses into an engineering process.
This guide covers how to build a useful test set, how to measure retrieval and generation separately so you can find the real source of a failure, how to use model-based judging without fooling yourself, and how to keep evaluating after launch. It fills in the detail behind the evaluation points in our [RAG development guide](/blog/rag-development-guide) and [LLM observability guide](/blog/llm-observability).
- Start with
- A test set of real questions with known good sources and answers
- Measure separately
- Did we retrieve the right context? Did we answer faithfully from it?
- LLM judges
- Useful, but calibrate against human-labelled examples
- After launch
- Keep the test set growing from real failures
Why one overall score is not enough
If a RAG system gives a wrong answer, there are two very different explanations. Either the retriever never surfaced the relevant passage, or it did and the model ignored, misread or embellished it. A single end-to-end quality number hides which one happened, and the fix is completely different: better chunking, hybrid search or reranking in the first case; better prompting, a stricter grounding instruction or a different model in the second.
So evaluate in two layers, and log enough on every query to attribute a failure to the right one.
Building a test set
A good test set is the most valuable evaluation asset you will own. It is built from the real world, not invented in a meeting:
- Collect real questions from users, support tickets, search logs or subject-matter experts. Include the awkward ones: vague phrasing, abbreviations, multi-part questions.
- Record the expected source (the document or passage that should be retrieved) and a short reference answer, written or approved by someone who knows the domain.
- Cover the categories that matter: easy lookups, multi-document questions, questions with no answer in the knowledge base (the system should say so), and questions where access permissions limit what may be returned.
- Start small and grow. Even a modest set run consistently is far better than none. Add a new case every time you find a real failure.
- Keep a held-out portion you do not tune against, so improvements are not just overfitting to the examples you stared at.
| Layer | Question it answers | Typical measures |
|---|---|---|
| Retrieval | Did the right passages come back? | Hit rate or recall of the expected source in the top results; rank of the first relevant result; precision of what was retrieved |
| Context quality | Is the retrieved context focused and non-redundant? | Share of retrieved text that is actually relevant; duplicate or near-duplicate chunks |
| Faithfulness (groundedness) | Is the answer supported by the retrieved context? | Share of claims in the answer that the context supports |
| Answer correctness | Is the answer actually right and complete? | Agreement with the reference answer, judged by humans or a calibrated judge |
| Abstention | Does it decline when the answer is not available? | Rate of correct refusals on unanswerable questions |
| Citations | Do cited sources really support the claims? | Citation accuracy against the retrieved passages |
Calibrate model-based judges before you trust them
Using an LLM to grade answers scales well but has known biases, such as favouring longer or more confident-sounding answers. Have people label a sample, compare the judge's grades to theirs, and refine the grading instructions until they agree acceptably. Re-check when you change the judge model. Treat judge scores as an instrument that needs calibration, not as ground truth.
Compare changes fairly
Every change (chunk size, embedding model, reranker, prompt, model version) should be evaluated against the same test set and compared with the previous baseline. Change one thing at a time where you can. Because generation can vary between runs, run enough cases that small random differences do not look like improvements, and look at individual regressions rather than only the average: a change that lifts the mean while breaking an important category of question is not a win.
Evaluation is not observability, but they feed each other
Offline evaluation tells you how the system performs on known cases before you ship. Production monitoring tells you about the cases you did not think of. The two connect: failures spotted in production (bad answers, negative user feedback, unanswered questions) should be turned into new test cases, as described in our LLM observability guide. Over time the test set becomes a record of everything your system has ever got wrong.
Evaluating hallucination risk specifically
Faithfulness and abstention tests are the practical way to measure the risk covered in our AI hallucinations guide. Include questions designed to tempt the model: ones where the knowledge base is silent, outdated or contradictory, and check that the system handles them honestly rather than inventing a plausible answer.
Security and access cases belong in the test set
For enterprise systems, include tests that confirm users cannot retrieve documents they are not permitted to see, and that instructions embedded in documents do not hijack the system. These are pass/fail checks rather than quality scores, and they should block a release if they fail.
- A test set built from real questions, with expected sources and reference answers
- Cases for easy lookups, multi-document questions and unanswerable questions
- A held-out portion not used for tuning
- Retrieval and generation measured separately
- Faithfulness and abstention tracked explicitly
- Any LLM judge calibrated against human-labelled samples
- Every change compared against the same baseline, with regressions inspected
- Production failures converted into new test cases
- Permission and injection tests treated as release blockers
Want RAG quality you can measure?
Talk to our AI engineering team about building an evaluation set and quality process for your RAG system.
Frequently asked questions
How do I evaluate a RAG system?+
Build a test set of real questions with expected sources and reference answers, then measure retrieval quality and answer quality separately so you can tell whether a failure came from the retriever or from generation.
What metrics matter most for RAG?+
Retrieval recall or hit rate of the expected source, faithfulness of the answer to the retrieved context, answer correctness, correct abstention on unanswerable questions and citation accuracy.
How big should a RAG test set be?+
Start with whatever you can build from real questions, even a modest set, and grow it every time you find a failure. A small set used consistently beats a large set built once and ignored.
Can I use an LLM to grade RAG answers?+
Yes, but calibrate it against human-labelled samples first, because model judges have biases such as favouring longer, more confident answers. Re-check calibration when the judge model changes.
Why separate retrieval evaluation from answer evaluation?+
Because the fixes differ. Retrieval failures call for chunking, search or reranking changes; generation failures call for prompting, grounding or model changes.
Should evaluation continue after launch?+
Yes. Turn real production failures into new test cases and re-run the set whenever data, prompts or models change, so quality does not drift unnoticed.
Written by
CodeSurge AI Engineering Team
The CodeSurge AI team designs and builds AI systems, SaaS products and enterprise integrations for clients in India, the UAE and beyond — this section shares the architecture patterns, cost drivers and implementation tradeoffs we work through on real projects.