Most RAG systems are evaluated with vibes. An engineer runs ten queries, eyeballs the results, and declares the system “working.” Three months later, a customer reports that the system confidently returned an answer that contradicted the source document. The engineer investigates and discovers that the retrieval pipeline has been returning irrelevant chunks for weeks. The generation model has been compensating by ignoring the retrieved context and answering from its training data. The system looked like it was working. It was not.

RAG evaluation needs to be decomposed into its constituent parts. A RAG system has two stages — retrieval and generation — and each stage has distinct failure modes that require distinct evaluation approaches. Evaluating end-to-end accuracy alone tells you the system is wrong but not why. Evaluating each stage separately tells you where to fix it.

This framework defines the metrics for each stage, the evaluation process, and the thresholds that indicate a healthy system. It is designed to be implemented with tools you already have: a labeled test set, a scoring script, and a monitoring pipeline.

Prerequisites

You need a test set of at least fifty queries that represent your production query distribution. Each query needs a labeled set of relevant documents or chunks from your corpus. This is the expensive part — labeling relevance requires domain expertise. Budget one to two days of domain expert time to build the initial test set.

You need your RAG system instrumented to log intermediate outputs: the retrieved chunks (with scores), the final prompt (with context), and the generated response. Without intermediate logging, you can only evaluate the final output, which makes root cause analysis impossible.

You need a judge — either a human evaluator or a judge model — that can assess response quality on defined criteria. Judge models (using an LLM to evaluate LLM outputs) are faster and cheaper than human evaluation but less reliable for nuanced judgments. Start with a judge model and validate its assessments against human evaluation on a sample.

The evaluation framework

Stage 1: Retrieval evaluation

Retrieval evaluation answers the question: did the system find the right context? This is the first gate. If retrieval fails, generation will fail regardless of model quality.

Retrieval precision. Of the chunks retrieved, what fraction are actually relevant to the query? A system that retrieves ten chunks but only three are relevant has a precision of 30%. Precision matters because irrelevant chunks in the context window confuse the generation model, consume tokens, and can cause the model to generate incorrect answers based on irrelevant information.

Target: precision above 70% at your standard retrieval depth. If your system retrieves ten chunks, at least seven should be relevant.

Retrieval recall. Of all the relevant chunks in the corpus, what fraction did the system retrieve? A system with high recall finds most of the relevant information. A system with low recall misses relevant context, forcing the generation model to answer without complete information.

Target: recall above 80%. This is harder to measure because you need to know all relevant chunks in the corpus, not just the ones retrieved. For smaller corpora, label all relevant chunks per query. For larger corpora, estimate recall by sampling.

Mean reciprocal rank (MRR). What is the position of the first relevant result? MRR measures whether the most relevant information appears early in the retrieval list. A system with high MRR puts the best context first, which helps the generation model focus on the most relevant information.

Target: MRR above 0.7. This means the first relevant result is, on average, in the top two positions.

Retrieval latency. How long does retrieval take? This is an operational metric, not a quality metric, but it directly affects user experience. A retrieval system that takes three seconds to find the right context is too slow for interactive applications.

Target: p95 retrieval latency under 500ms for interactive applications, under five seconds for batch processing.

Stage 2: Context evaluation

Between retrieval and generation sits a quality gate that most teams skip: evaluating whether the retrieved context is sufficient to answer the query.

Context sufficiency. Given the retrieved chunks, can a knowledgeable human answer the query? If a human cannot answer the question from the retrieved context, the model cannot either. This check catches retrieval failures that pass precision metrics — a system can retrieve technically relevant chunks that still lack the specific information the query needs.

Evaluate context sufficiency using your judge (human or model). For each query, ask: “Is the retrieved context sufficient to fully answer this query?” Record the yes/no rate.

Target: context sufficiency above 85%.

Context redundancy. Do the retrieved chunks contain duplicate or near-duplicate information? Redundant chunks waste context window space and can cause the generation model to overweight repeated information. Measure redundancy as the average semantic similarity between retrieved chunks. High redundancy indicates poor chunking strategy.

Target: average pairwise similarity below 0.85 among retrieved chunks.

Context coherence. Do the retrieved chunks form a coherent picture when read together? Chunks pulled from different sections of different documents may each be individually relevant but collectively incoherent. A judge can assess coherence subjectively.

Stage 3: Generation evaluation

Generation evaluation answers the question: did the model produce a correct, complete, and well-grounded response given the retrieved context?

Faithfulness. Does the response accurately reflect the retrieved context? Faithfulness measures hallucination relative to the context. A response that contradicts or fabricates information not present in the retrieved chunks fails the faithfulness check.

This is the most critical generation metric. A faithful response that is incomplete is better than a complete response that hallucinates. Users can seek more information. They cannot recover from acting on false information.

Measure faithfulness by checking each factual claim in the response against the retrieved chunks. The claim should be directly supported by the context, not merely consistent with it.

Target: faithfulness above 95%. Below this threshold, the system is producing enough hallucinated content to erode user trust.

Answer relevance. Does the response address the query? A response can be faithful to the context but irrelevant to the question. This happens when retrieval returns relevant chunks for a different interpretation of the query, and the generation model answers based on those chunks.

Target: answer relevance above 90%.

Answer completeness. Does the response cover all aspects of the query? Multi-part queries require multi-part answers. A response that addresses only one part of a two-part query is incomplete even if the addressed part is correct.

Target: completeness above 80%. This is the most subjective metric and benefits most from human evaluation.

Answer conciseness. Does the response include unnecessary information? Conciseness is the least critical metric but matters for user experience. A response that buries the answer in three paragraphs of context is correct but frustrating.

Target: judge-model rating of “appropriately concise” above 75%.

Stage 4: End-to-end evaluation

End-to-end evaluation measures the system as a whole. It is the last stage because root cause analysis requires the per-stage metrics.

End-to-end accuracy. For queries with a known correct answer, what percentage of responses are correct? This is the metric most teams start with, and it is the least useful for debugging. A low accuracy score tells you the system is failing but not whether the failure is in retrieval, context assembly, or generation.

Target: depends entirely on the use case. Define your accuracy requirement based on the cost of incorrect answers.

User satisfaction. For user-facing systems, measure whether users found the response helpful. This can be a thumbs-up/thumbs-down widget, a satisfaction survey, or a follow-up question rate (if users ask follow-up questions, the first answer was probably incomplete or unclear).

Target: satisfaction above 80%.

End-to-end latency. Total time from query submission to response delivery. This is the sum of retrieval latency, context assembly latency, and generation latency. It determines whether the system is usable for your application.

Target: under three seconds for interactive applications.

Evaluation process

Run the full evaluation suite weekly against your test set. Track metrics over time, not as point-in-time snapshots. Metric trends reveal problems that single evaluations miss: a slow decline in retrieval recall over six weeks indicates an index degradation problem that a single evaluation would not catch.

When a metric drops below its threshold, diagnose using the per-stage metrics. If retrieval metrics are stable but generation metrics dropped, the problem is in the generation model or the prompt. If retrieval metrics dropped, the problem is in the index, the embedding model, or the data ingestion pipeline.

After any change to the RAG system — new data added, embedding model changed, prompt modified, retrieval parameters adjusted — run the evaluation suite before deploying to production. This is the CI/CD gate for RAG systems.

Common failure modes

Evaluating only end-to-end. End-to-end accuracy is necessary but insufficient for debugging. Without per-stage metrics, a 10% accuracy drop could be a retrieval problem, a generation problem, or a context assembly problem. You would have to investigate all three.

Using too-small test sets. Ten queries do not give you statistically significant results. Fifty queries give you directional confidence. One hundred or more give you actionable precision. Grow your test set over time by adding queries that exposed failures in production.

Static test sets. A test set that never changes stops finding new failure modes. Add production queries to the test set quarterly, especially queries that caused user complaints or manual overrides.

Ignoring latency. A system that produces correct answers in eight seconds is not useful for interactive applications. Evaluate latency alongside quality. A 5% accuracy reduction that cuts latency in half may be the right tradeoff.

Next step

Build your test set. Take fifty real queries from your production logs or your stakeholder interviews. For each query, label the relevant chunks in your corpus. This is the most labor-intensive step and the most valuable. Without labeled data, every other evaluation step is guesswork. Start today, and aim to have the test set ready within one week.