LLM Evaluation — RAGAS, DeepEval, G-Eval

Your RAG system answers "June 29th, 2007." The gold reference says "June 29, 2007." Exact Match scores 0. F1 scores about 75%. A human reading both would call it 100% correct.

Now multiply that gap by 10,000 test cases, then again by every change you make to the retriever, the chunking, the prompt, or the model itself. What you need is an evaluator that understands meaning, runs cheaply at scale, doesn't lie about regressions, and points at the right failure mode when something breaks.

2026 settled on three frameworks for this job.

• RAGAS (Retrieval-Augmented Generation ASsessment). Four RAG metrics (faithfulness, answer relevance, context precision, context recall) backed by NLI and LLM-judge scoring. Research-grade, lightweight.
• DeepEval. Pytest for LLMs: G-Eval, task-completion, hallucination, and bias metrics, built to live in CI/CD.
• G-Eval. A method, and a DeepEval metric: LLM-as-judge scoring with chain-of-thought reasoning, a custom rubric, and a 0-1 score.

All three lean on LLM-as-judge underneath. This lesson builds intuition for the method itself, then the trust layer you need before you let it gate a merge.

LLM-as-judge means swapping a static metric for an LLM that scores outputs against a rubric. Feed it (query, context, answer) and prompt a judge model: "Score 0-1 on faithfulness." It hands back the number.

It works because LLMs approximate human judgment for a fraction of the cost: GPT-4o-mini runs about $0.003 per scored case, so a 1000-sample regression eval costs under $5.

It fails silently in three specific ways:

1. Judge bias. Judges favor longer answers, answers from their own model family, and answers that match the prompt's style.
2. JSON parsing failures. Bad JSON turns into a NaN score, which gets silently dropped from the aggregate. RAGAS users know this pain well. Gate it with try/except and an explicit failure mode instead of letting it vanish.
3. Drift across model versions. Upgrade the judge and every metric moves with it. Freeze the judge model and its version.

The RAG four. One term worth pinning down before the table: NLI (natural language inference) classifies whether one piece of text entails, contradicts, or is neutral toward another. Faithfulness runs on exactly that classifier.

G-Eval turns a custom quality criterion, something like "did the answer cite the correct source," into a scored rubric. Hand it the criterion and the framework auto-expands it into chain-of-thought evaluation steps, then returns a 0-1 score. Reach for it when you need a domain-specific quality dimension RAGAS doesn't cover.

Calibration is the step that makes any of this trustworthy. Never take a raw judge score at face value until you've checked it against humans: hand-label 100 examples, plot judge scores against human scores, and compute Spearman rho (a rank-correlation score from -1 to 1). Below 0.7, the rubric needs work before it goes anywhere near a CI gate.

Step 1: faithfulness with NLI (RAGAS-style)

from typing import Callable
from transformers import pipeline
 
nli = pipeline("text-classification",
               model="MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli",
               top_k=None)
 
# `llm` is any callable: prompt str -> generated str.
# Example: llm = lambda p: client.messages.create(model="claude-haiku-4-5", ...).content[0].text
LLM = Callable[[str], str]
 
 
def atomic_claims(answer: str, llm: LLM) -> list[str]:
    prompt = f"""Break this answer into simple factual claims (one per line):
{answer}
"""
    return llm(prompt).splitlines()
 
 
def faithfulness(answer: str, context: str, llm: LLM) -> float:
    claims = atomic_claims(answer, llm)
    if not claims:
        return 0.0
    supported = 0
    for claim in claims:
        result = nli({"text": context, "text_pair": claim})[0]
        entail = next((s for s in result if s["label"] == "entailment"), None)
        if entail and entail["score"] > 0.5:
            supported += 1
    return supported / len(claims)

Break the answer into atomic claims, then run each one against the retrieved context with NLI. Faithfulness is just the fraction that holds up.

Step 2: answer relevance

import numpy as np
from sentence_transformers import SentenceTransformer
 
# encoder: any model implementing .encode(texts, normalize_embeddings=True) -> ndarray
# e.g., encoder = SentenceTransformer("BAAI/bge-small-en-v1.5")
 
def answer_relevance(question: str, answer: str, encoder, llm: LLM, n: int = 3) -> float:
    prompt = f"Write {n} questions this answer could be the answer to:\n{answer}"
    generated = [line for line in llm(prompt).splitlines() if line.strip()][:n]
    if not generated:
        return 0.0
    q_emb = np.asarray(encoder.encode([question], normalize_embeddings=True)[0])
    g_embs = np.asarray(encoder.encode(generated, normalize_embeddings=True))
    sims = [float(q_emb @ g_emb) for g_emb in g_embs]
    return sum(sims) / len(sims)

If the answer implicitly answers a different question than the one asked, relevance drops.

Step 3: G-Eval custom metric

from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCaseParams, LLMTestCase
 
metric = GEval(
    name="Correctness",
    criteria="The answer should be factually accurate and match the expected output.",
    evaluation_steps=[
        "Read the expected output.",
        "Read the actual output.",
        "List factual claims in the actual output.",
        "For each claim, mark supported or unsupported by the expected output.",
        "Return score = fraction supported.",
    ],
    evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.EXPECTED_OUTPUT],
)
 
test = LLMTestCase(input="When was the first iPhone released?",
                   actual_output="June 29th, 2007.",
                   expected_output="June 29, 2007.")
metric.measure(test)
print(metric.score, metric.reason)

The evaluation steps are the rubric. Explicit steps are more stable than implicit "score 0-1" prompts.

Step 4: CI gate

import deepeval
from deepeval.metrics import FaithfulnessMetric, ContextualRelevancyMetric
 
 
def test_rag_system():
    cases = load_regression_cases()
    faith = FaithfulnessMetric(threshold=0.85)
    rel = ContextualRelevancyMetric(threshold=0.7)
    for case in cases:
        faith.measure(case)
        assert faith.score >= 0.85, f"faithfulness regression on {case.id}"
        rel.measure(case)
        assert rel.score >= 0.7, f"relevancy regression on {case.id}"

Ship as a pytest file. Run on every PR. Block merges on regressions.

Step 5: toy eval from scratch

See code/main.py. Stdlib-only approximations of faithfulness (overlap of answer claims with context) and relevance (overlap of answer tokens with question tokens). Not production. Shows the shape.

• No calibration. A judge with 0.3 correlation to human labels is noise, not signal. Require a calibration run before shipping.
• Self-evaluation. Using the same LLM to generate and judge inflates scores by 10-20%. Use a different model family for the judge.
• Positional bias in pairwise judging. Judges prefer whichever option is presented first. Always randomize the order and run both.
• Raw aggregate hides failures. A mean of 0.85 can hide a 5% catastrophic-failure rate. Always inspect the bottom quantile.
• Golden dataset rot. Unversioned eval sets drift over time and quietly break longitudinal comparison. Tag the dataset with every change.
• LLM cost. At scale, judge calls dominate the bill. Use the cheapest model that clears your calibration threshold: GPT-4o-mini, Claude Haiku, Mistral-small.

Here's where each framework actually earns its place in the 2026 stack:

A typical setup runs RAGAS for monitoring, DeepEval for CI, and G-Eval for the quality dimensions neither one covers. Run all three; they disagree usefully.

This lesson ships as outputs/skill-eval-architect.md. Hand it a use case and it returns a full eval plan: metrics, judge model, calibration target, dataset versioning, and a CI gate, with hard refusals baked in for uncalibrated judges, self-evaluation, and aggregate-only reporting.

---
name: eval-architect
description: Design an LLM evaluation plan with calibrated judge and CI gates.
version: 1.0.0
phase: 5
lesson: 27
tags: [nlp, evaluation, rag]
---
 
Given a use case (RAG / agent / generative task), output:
 
1. Metrics. Faithfulness / relevance / context-precision / context-recall + any custom G-Eval metrics with criteria.
2. Judge model. Named model + version, rationale for cost vs accuracy.
3. Calibration. Hand-labeled set size, target Spearman rho vs human > 0.7.
4. Dataset versioning. Tag strategy, change log, stratification.
5. CI gate. Thresholds per metric, regression-window logic, bottom-quantile alert.
 
Refuse to rely on a judge untested against ≥50 human-labeled examples. Refuse self-evaluation (same model generates + judges). Refuse aggregate-only reporting without bottom-10% surfacing. Flag any pipeline where judge upgrade lands without parallel baseline eval.

1. Easy. Use RAGAS on 10 RAG examples with known hallucinations. Verify the faithfulness metric catches each one.
2. Medium. Hand-label 50 QA answers 0-1 for correctness. Score with G-Eval. Measure Spearman rho between judge and human.
3. Hard. Build a pytest CI gate with DeepEval. Intentionally regress the retriever. Verify the gate fails. Add bottom-quantile alerting via threshold check on the lowest 10%.

• Es et al. (2023). RAGAS: Automated Evaluation of Retrieval Augmented Generation: the RAGAS paper.
• Liu et al. (2023). G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment: the G-Eval paper.
• DeepEval docs: open production stack.
• Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena: biases, calibration, limits.
• MLflow GenAI Scorer: unifying framework that integrates RAGAS, DeepEval, Phoenix.