RAG Evaluation: Precision, Recall, MRR, nDCG, Faithfulness, Answer Relevance

A RAG system has at least four moving parts: chunker, retriever, reranker, generator. Any of them can be the reason an answer is wrong, and without per-stage metrics you are guessing which one.

Say a user reports a bad answer. Maybe the chunker cut the answer span in half. Maybe the retriever left the right chunk out of the top-k. Maybe the reranker pushed it below position one, or the generator ignored it and made something up. The answer text alone can't tell you which. You need four kinds of signal:

• Retrieval metrics grade what came out of the retriever.
• Ranking metrics grade where the right chunk sat in the order.
• Faithfulness grades whether the generator stayed inside the retrieved context.
• Answer relevance grades whether the answer addresses the question at all.

The code builds all six metrics on a fixture qrels file. The eval runs offline and deterministically, using a mock judge. In production you swap in a real LLM-as-judge.

Precision@k

Precision@k is the fraction of the top-k retrieved documents that are in the gold set. If gold has three documents and the top-3 contains two of them plus one wrong one, precision@3 is 2 / 3. Reach for it when an irrelevant chunk is expensive: the generator burns tokens on it, or the chunk poisons the answer.

Recall@k

Recall@k is the fraction of the gold documents that appear in the top-k. If gold has three documents and the top-5 contains all three, recall@5 is 1.0. Reach for it when a missed answer is expensive and you'd rather show one extra wrong chunk than lose the answer chunk.

Recall@k is the number people usually quote in production RAG. A generator can usually drop an irrelevant chunk, but it can't build an answer from a chunk that was never retrieved.

MRR (Mean Reciprocal Rank)

MRR is the mean, across queries, of 1 / position of the first relevant document in the ranked list. It is a single-number summary of how reliably the retriever puts a good answer at the top.

Position one dominates. A gold doc at rank 1 contributes 1.0, rank 2 contributes 0.5, rank 10 contributes 0.1, so the score mostly reflects the head of the list.

nDCG@k

nDCG@k is normalized discounted cumulative gain: the DCG of your ranking divided by the ideal DCG, the score you'd get by ranking perfectly. The result runs from 0 to 1. DCG assigns a gain to each retrieved document, discounts it by the log of its position, and sums.

nDCG also handles graded relevance. The gold can say "doc A is 3, doc B is 2, doc C is 1", where MRR and recall@k flatten everything to relevant or not. Use nDCG when a query has several partially relevant documents.

Faithfulness

Faithfulness is the fraction of claims in the generated answer that the retrieved context supports. The standard implementation prompts an LLM-as-judge with (claim, context) and gets back yes or no for each claim. You may also see it called groundedness, support, or attribution.

It catches the generator that invents content. The retriever can return exactly the right chunks and the system is still broken if the generator hallucinates on top of them.

The code uses a deterministic mock judge: a claim counts as supported when the share of its content tokens that also appear in the retrieved context reaches a threshold (0.4 by default). A real model call replaces the judge in production, and the shape of the metric stays the same.

Answer relevance

Answer relevance asks whether the answer addresses the question. Faithfulness asks whether the answer is grounded in the context. The two come apart: a faithful but off-topic answer scores high on faithfulness and low on relevance, and a short on-topic answer that ignores the context does the opposite.

The standard implementation is also an LLM-as-judge over (question, answer). The code substitutes a token-overlap check for it, scored as 1.0 or 0.0 per query.

{
  "qid": "q1",
  "query": "what is the threshold before a multipart upload is dropped",
  "gold_doc_ids": ["d3", "d1"],
  "gold_answer_substring": "three failed parts",
  "graded_relevance": {"d3": 3, "d1": 2, "d2": 1},
}

Each query carries:

• the query string,
• a set of gold doc ids (for precision / recall / MRR),
• a graded relevance dict (for nDCG),
• the gold answer substring (kept as reference metadata on each qrel; faithfulness here is computed by judging extracted claims against the retrieved context, not against this substring).

The query says "dropped" while the corpus says "aborted", which is exactly the vocabulary gap a plain term-overlap baseline cannot bridge. The fixture has four queries over a twelve-document corpus.

In production you label these. The fixture here is hand-built so the eval runs out of the box.

code/main.py implements:

• precision_at_k(retrieved, gold, k) - hits in the top-k divided by k, even when the retriever returns fewer than k documents.
• recall_at_k(retrieved, gold, k) - hits in the top-k divided by the number of gold documents.
• mean_reciprocal_rank(retrieved_per_query, gold_per_query) - the mean over queries of reciprocal_rank.
• ndcg_at_k(retrieved, graded_relevance, k) - DCG / IDCG with graded gains only: each document contributes (2**rel - 1) / log2(rank + 1). There is no binary mode, so every gold document needs a grade.
• extract_claims(answer) - splits an answer into sentence-shaped claims.
• faithfulness(claims, context_texts, judge) - fraction of claims judged supported.
• answer_relevance(question, answer, judge) - judge on whether the answer addresses the question.
• MockJudge - deterministic token-overlap judge so the eval runs offline.
• evaluate_pipeline(pipeline_fn, qrels, ks) - the orchestrator that runs all the metrics.
• A demo that runs three pipeline variants against the qrels and prints a metrics table: a bag-of-words term-overlap baseline over whole documents (the demo has no chunker), a hybrid that adds a synonym pass and a title bonus, and hybrid plus a title-match rerank.

Run it:

python3 code/main.py

The output is one metrics table with precision@k, recall@k, nDCG@k, MRR, faithfulness, and answer relevance for each variant. Read it carefully, because it is less flattering than a demo usually is:

• Hybrid beats the baseline only at the top of the list: recall@1 is 0.875 vs 0.375, nDCG@1 is 1.000 vs 0.500, and MRR is 1.000 vs 0.750. Recall@3 and recall@5 are 1.000 for all three variants.
• Hybrid plus rerank equals hybrid on every metric, because hybrid already ranks a gold document first for all four queries. The fixture cannot show a rerank gain.
• Faithfulness is 1.000 everywhere because each answer is built by concatenating retrieved document bodies, so every claim overlaps its context by construction.

LLM-as-judge bias. A model tends to rate its own outputs as more faithful than they are. Use a different model family for the judge than for the generator, or hand-grade a sample.

Qrels rot. Gold answers drift as the corpus changes. A doc that was gold for q1 in January 2024 may be wrong by October 2024 because the team renamed the function. Schedule a quarterly qrels review.

Faithfulness micro-checks miss macro-claims. Per-sentence faithfulness can pass while the overall structure of the answer misleads. Add a sampled qualitative review on top of the automated metric.

Recall@k masks per-query failures. A 90% average recall can hide a query class that keeps missing. Slice the qrels by query class (literal, paraphrased, multi-topic) and report per slice.

Production patterns:

• Run the eval on each retriever or generator change, and treat a recall@k regression like a failing test.
• Persist the metric trace per query. When a user complains, find the matching qrels entry and check whether the eval would have caught it.
• Tier the qrels: a smoke set of 20 queries in CI, a regression set of 200 nightly, a deep set of 2000 weekly.

Lesson 69 wires the full pipeline (chunker, retriever, reranker, generator) together and runs this eval against the end-to-end system.

1. Add a fifth retrieval metric: hit-rate@k. Compare it against recall@k. Explain when they differ.
2. Implement a graded faithfulness: 0 (unsupported), 1 (partially supported), 2 (fully supported). Update the metric accordingly.
3. Replace the mock judge with a real model call. Measure the disagreement between the mock and the real judge on the fixture.
4. Add a query-class slice ("literal", "paraphrased", "multi-topic"). Report per-slice metrics.
5. Add an "answer length" metric and correlate it with faithfulness. Plot the curve.

• Buckley, Voorhees, "Evaluating Evaluation Measure Stability", SIGIR 2000 - the canonical paper on ranking metrics
• Jarvelin, Kekalainen, "Cumulated Gain-based Evaluation of IR Techniques" - the nDCG paper
• Ragas: Automated Evaluation of RAG Pipelines
• Anthropic, Evaluating RAG
• Phase 11 lesson 10 - evaluation framework foundations
• Phase 19 lessons 64-67 - components evaluated here
• Phase 19 lesson 69 - the end-to-end pipeline this eval grades