Capstone Lesson 41: Full Evaluation Pipeline

An eval is a function that scores a model against a held-out dataset and returns a number. One eval doesn't describe a language model. Perplexity says how well the model fits the language distribution, and nothing about whether it answers questions. Exact-match says whether the model produced the gold string, and punishes correct paraphrases. Token F1 forgives paraphrase, but lexical overlap with wrong content can fool it. LLM-as-judge captures qualitative dimensions, at the cost of being expensive and stochastic.

You want all four. Each covers a dimension the others miss, and each runs on a held-out subset shaped for its metric. The final report puts the per-task numbers side by side with an aggregate, so a reviewer can see which trade-offs the model is making.

This lesson builds that pipeline in one file.

Each eval is a function that takes a model, a tokenizer, and a dataset and returns an EvalResult. The result carries the metric value, per-example records for inspection, and a name for the aggregate. run_demo runs the four evals and hands their results to aggregate, which takes an optional weights dict.

Perplexity is exp(mean negative log-likelihood per token). Two traps sit in the implementation:

• The mean has to run over actual token positions, not batch * sequence. Padding tokens must stay out of the denominator, or perplexity looks better than it is.
• The model predicts the next token, so logits at position i predict the token at position i+1. An off-by-one here is silent: the loss still trains, but the metric means nothing.

The eval sums -log p(token) over non-pad positions, tracks a token count per batch, and divides once at the end. Averaging per-batch perplexities instead would under-weight short sequences. The sum-then-divide version is also the textbook definition.

Exact-match checks whether the prediction equals the reference after both are normalised:

• Lowercase.
• Strip surrounding whitespace.
• Collapse internal whitespace runs to a single space.
• Drop trailing terminal punctuation (., !, ?) from both sides, unconditionally.

Normalisation is what makes the metric usable. A model that says "Paris" is right, and so is "Paris.", and so is " paris ". After normalisation the answer still has to be the same string.

Token F1 is the harmonic mean of precision and recall, computed over the bag of tokens. The steps:

1. Normalise prediction and reference: lowercase, strip, collapse whitespace. Punctuation is not stripped here, so Paris. and Paris count as different tokens.
2. Split each into a list of tokens (whitespace tokenisation).
3. Count the multiset intersection.
4. Precision = intersection_count / len(pred_tokens). Recall = intersection_count / len(ref_tokens). F1 is their harmonic mean.

If both prediction and reference are empty, F1 is 1 (a vacuous match). If only one is empty, F1 is 0. This follows the SQuAD evaluation reference and gives stable numbers across paraphrases.

A judge is a scorer that takes an instruction, a model's prediction, and a reference, and returns a score plus a rationale. A real one is a frontier model behind an API. This lesson has to run offline, so the mock judge is a deterministic function that returns a score in {1, 2, 3, 4, 5} and a one-line rationale. The rules are explicit:

• 5 if normalised prediction equals normalised reference.
• 4 if token F1 between prediction and reference is at least 0.8.
• 3 if token F1 is in [0.5, 0.8).
• 2 if token F1 is in [0.2, 0.5).
• 1 otherwise.

It isn't a real judge, but it has the right interface. To swap in a real model, change one function. The pipeline doesn't care which one it gets.

The aggregate is a weighted mean of normalised eval scores. Each eval reports its own number, mapped into [0, 1]:

• Perplexity: normalise as 1 / (1 + log(perplexity)). A perplexity of 1 maps to 1, and infinity maps to 0.
• Exact-match: already in [0, 1].
• Token F1: already in [0, 1].
• Judge: divide by 5.

Weights are configurable. aggregate renormalises them over the evals actually present, so running a subset still gives a weighted mean in [0, 1]. normalise_metric also maps a perplexity of zero or below, infinity, or NaN to 0. The default mix is 0.2 perplexity, 0.3 exact-match, 0.3 token F1, 0.2 judge. Picking weights is a product decision, so the knob is exposed for you to experiment with.

There is no suite or aggregator class. Each eval is a free function: perplexity_eval(model, tok, texts, max_len, batch_size) for perplexity, and (model, tok, pairs) for the other three (the judge eval also takes an optional judge_fn). Each returns an EvalResult. The free function aggregate collects the results into a FinalReport, and run_demo orchestrates the whole run. The demo prints the table and writes a JSON copy that downstream CI can ingest.

The implementation is one main.py plus tests.

1. TinyGPT: the same decoder-only architecture used in lessons 38-40, included so the lesson stands alone.
2. InstructionTokenizer: byte tokeniser with INST / RESP / PAD specials.
3. Four fixtures: an LM corpus, an EM set, an F1 set, and a judge set. Twenty examples each, deterministic.
4. perplexity_eval: returns EvalResult with the perplexity value, a per-example record for each text (score is that text's perplexity), and total_tokens and mean_nll as extras.
5. exact_match_eval: returns mean EM and per-example records.
6. token_f1_eval: returns mean token F1 and per-example records.
7. mock_judge and judge_eval: per-example score and rationale, mean score across the set.
8. normalise_metric: per-eval normalisation rule.
9. aggregate: weighted mean, returned as a FinalReport dataclass.
10. run_demo: trains a tiny model briefly, runs all four evals, prints the report table, writes the JSON, and exits zero on success.

The report has three layers. The aggregate score is on top. The four per-eval numbers sit below it, and the per-example breakdowns for diagnostics sit below those. A failing CI run usually wants the aggregate. A reviewer chasing a regression wants the per-example layer, to see which inputs the model got wrong.

The JSON dump uses stable keys, so a CI dashboard can plot trend lines across versions. The pretty-printed table is for the human at the terminal after a training run.

• Add a calibration eval: do the model's softmax probabilities match its accuracy? Bucket predictions by confidence and report the empirical accuracy per bucket.
• Add a robustness eval: tag each example with a perturbation (typo, paraphrase, distractor) and report the metric drop per perturbation.
• Replace the mock judge with a real model behind an HTTP call. The function signature stays the same.
• Add per-task weight learning: instead of fixed weights, fit weights to a target preference order over models.

You now have the four evals, the aggregator, and the report. Real evaluation pipelines layer more dimensions on top, and the pattern holds: one function per eval, one aggregator, one report.