The Evaluation Problem
You built a RAG system. It retrieves documents. It generates answers. But how do you know if it is actually good?
Most teams measure retrieval precision and call it a day. But retrieval is only half the story. A RAG system can retrieve perfect documents and still generate terrible answers.
We learned this the hard way when our "high-precision" RAG system started confidently making up facts that were nowhere in the retrieved context.
The RAG Evaluation Framework
We evaluate RAG systems across four dimensions:
| Dimension | What It Measures | Key Metrics |
|---|---|---|
| Retrieval Quality | Are we finding the right documents? | Precision, Recall, MRR, NDCG |
| Context Relevance | Is the context useful for answering? | Context Precision, Context Recall |
| Faithfulness | Does the answer stick to the context? | Hallucination Rate, Attribution Score |
| Answer Quality | Is the answer actually correct and helpful? | Correctness, Completeness, Coherence |
Dimension 1: Retrieval Quality
The foundation. If retrieval fails, everything else fails.
Standard Metrics
These are well-known but worth implementing correctly:
| Metric | Formula | What It Tells You |
|---|---|---|
| Precision@K | Relevant in top K / K | Quality of top results |
| Recall@K | Relevant in top K / Total relevant | Coverage of relevant docs |
| MRR | 1 / Rank of first relevant | How quickly you find something relevant |
| NDCG | Normalized discounted cumulative gain | Ranking quality with graded relevance |
Implementation
Our retrieval evaluator:
| Metric | Our Score | Industry Benchmark |
|---|---|---|
| Precision@5 | 0.82 | 0.70-0.85 |
| Recall@10 | 0.89 | 0.75-0.90 |
| MRR | 0.79 | 0.65-0.80 |
| NDCG@10 | 0.84 | 0.70-0.85 |
Dimension 2: Context Relevance
Retrieved documents might match the query but not help answer it.
Context Precision
What fraction of retrieved context is actually useful?
We found that 23% of our retrieved chunks were topically related but did not contain information needed to answer the question. High retrieval precision, low context precision.
Context Recall
Does the retrieved context contain all the information needed to fully answer the question?
This requires ground-truth annotations of what information should be present.
Dimension 3: Faithfulness (The Hallucination Problem)
This is where most RAG systems fail silently. The model generates fluent, confident text that contradicts or extends beyond the context.
Types of Unfaithfulness
| Type | Description | Example |
|---|---|---|
| Contradiction | Answer contradicts context | Context: "Released in 2023" Answer: "Released in 2022" |
| Fabrication | Answer adds unsupported facts | Context mentions X, answer adds detail Y not in context |
| Extrapolation | Answer extends beyond context | Context has partial info, answer fills gaps with assumptions |
Measuring Faithfulness
We use an LLM-as-judge approach with careful prompting:
Our prompt template for faithfulness evaluation asks the evaluator to check if every claim in the answer is supported by the context, identify any claims not in the context, and check for contradictions.
Faithfulness Results
| Category | Faithfulness Score | Notes |
|---|---|---|
| Factual queries | 94% | High faithfulness |
| Analytical queries | 82% | More extrapolation |
| Comparison queries | 78% | Often adds unsupported comparisons |
| Creative queries | 65% | Highest hallucination risk |
Dimension 4: Answer Quality
Even faithful answers can be unhelpful. We evaluate three aspects:
Correctness
Does the answer correctly convey the information from the context?
This catches cases where the model misinterprets correct context.
Completeness
Does the answer address all aspects of the question?
We found our system often answered the first part of multi-part questions and ignored the rest.
Coherence
Is the answer well-structured and easy to understand?
A correct, complete answer can still be confusing if poorly organized.
Quality Rubric
| Score | Correctness | Completeness | Coherence |
|---|---|---|---|
| 5 | Fully correct | All aspects addressed | Clear and well-organized |
| 4 | Minor errors | Most aspects addressed | Generally clear |
| 3 | Some errors | Half addressed | Somewhat confusing |
| 2 | Significant errors | Few aspects addressed | Hard to follow |
| 1 | Mostly incorrect | Question not addressed | Incoherent |
Automated Evaluation Pipeline
Manual evaluation does not scale. Here is our automated pipeline:
The pipeline runs on every RAG change with a test set of 500 queries with ground-truth answers. It evaluates all four dimensions and compares against baseline. Changes that regress any metric by more than 2% are flagged for review.
Continuous Monitoring
In production, we sample 1% of queries for evaluation and track metrics over time, alerting on metric degradation.
Building Your Evaluation Dataset
The hardest part of RAG evaluation is creating good test data.
Dataset Requirements
| Requirement | Why It Matters |
|---|---|
| Representative queries | Cover real usage patterns |
| Ground-truth answers | Know what correct looks like |
| Relevance judgments | Know which docs should be retrieved |
| Edge cases | Test failure modes |
Our Dataset Composition
| Category | % of Dataset | Examples |
|---|---|---|
| Simple factual | 30% | "What is the API rate limit?" |
| Multi-hop | 20% | "How does X compare to Y?" |
| Analytical | 15% | "Why did we choose this approach?" |
| Temporal | 10% | "What changed in version 2.0?" |
| Negative | 10% | Questions with no answer in corpus |
| Adversarial | 15% | Edge cases, ambiguous queries |
Creating Ground Truth
For each query, we annotate:
- •Relevant documents (with relevance grades 0-3)
- •Ideal answer (written by domain expert)
- •Key facts that must be present
- •Facts that must NOT be hallucinated
Key Metrics Dashboard
What we track daily:
| Metric | Target | Current | Trend |
|---|---|---|---|
| Retrieval Precision@5 | >0.80 | 0.82 | Stable |
| Context Relevance | >0.75 | 0.77 | Improving |
| Faithfulness | >0.90 | 0.88 | Watch |
| Answer Correctness | >0.85 | 0.86 | Stable |
| Answer Completeness | >0.80 | 0.79 | Improving |
Lessons Learned
- 1Retrieval metrics are necessary but not sufficient - Good retrieval does not guarantee good answers.
- 2Faithfulness is your biggest risk - Users trust your system. Hallucinations destroy that trust.
- 3Automate evaluation early - Manual evaluation does not scale. Build the pipeline before you need it.
- 4Test negative cases - Knowing when NOT to answer is as important as answering correctly.
- 5Track metrics over time - Point-in-time evaluation misses degradation.
- 6Human evaluation for calibration - Periodically validate automated metrics against human judgment.
RAG evaluation is not glamorous, but it is the difference between a demo and a production system. Measure what matters, catch problems early, and iterate continuously.