Classical Metrics

A classical metric is a deterministic function that scores a prediction against reference text by surface overlap, with no model in the loop. That makes the numbers cheap and reproducible, and it also makes them easy to misread.

You will see one paper report BLEU 28.3 and another report BLEU 0.283. You will see ROUGE-L scores that differ by ten points across two libraries because one lowercases and the other doesn't. Writing the metrics yourself is the quickest way out of that confusion: you can point at the line where the tokenizer is decided and the line where the smoothing is applied. Comparing numbers across papers then comes down to reading the metric setup instead of arguing about libraries.

Stdlib plus numpy is enough. BLEU is counting and a clamp. ROUGE-L is dynamic programming. F1 is an intersection of token counts. The hard part is picking a tokenizer and sticking with it.

The tokenizer is re.findall(r"\w+", text.lower()): lowercase the text, keep runs of alphanumeric characters, drop punctuation. F1, BLEU-4, and ROUGE-L share it, and the runner doesn't get a say. Exact match and accuracy skip it and compare stripped strings. Swap the tokenizer and you're running a different benchmark.

TOKEN_RE = re.compile(r"\w+", re.UNICODE)
def tokenize(text):
    return TOKEN_RE.findall(text.lower())

This is a deliberate simplification. Production setups have to deal with CJK, contractions, and code identifiers. What I want you to take from it is that the tokenizer is a contract, not a knob.

Exact match scores 1.0 when the prediction equals a target string after stripping whitespace, and 0.0 otherwise.

def exact_match(prediction, targets):
    if not targets:
        return 0.0
    pred = prediction.strip()
    return 1.0 if any(pred == t.strip() for t in targets) else 0.0

An empty target list scores 0.0. There is no lowercasing and no punctuation stripping, so "Paris." does not match "Paris".

The aggregate over a dataset is the mean of the per-task scores. It's the workhorse for arithmetic, MCQ, and short classification tasks.

Token-level F1 is the harmonic mean of token precision and recall between a prediction and a target. Build a token multiset for each. Precision is the size of the multiset intersection divided by the size of the prediction's multiset, and recall is the same intersection divided by the size of the target's. The implementation also covers the empty-prediction and empty-target cases.

For multi-target tasks, take the best F1 over the target list. That matches the SQuAD-style behaviour widely reported in the literature.

BLEU-4 scores a candidate by its modified n-gram precision for n = 1 through 4, combined by geometric mean and scaled by a brevity penalty. It's the canonical machine-translation metric and it still shows up in summarisation work. The version score("bleu_4", ...) dispatches to is sentence-level: bleu4(prediction, reference) scores one candidate against one reference, with the standard brevity penalty and additive-one smoothing on modified n-gram counts, so one missing 4-gram doesn't push the score to zero. When a task has several targets, score takes the max over them. main.py also has a corpus_bleu that pools clipped counts and lengths across a dataset before taking the geometric mean, but the dispatcher doesn't call it.

Modified precision clips each candidate n-gram count by the count of that n-gram in the reference (one reference per bleu4 call), so a candidate can't inflate its score by repeating one phrase. For each candidate-reference pair you compute that precision at n = 1, 2, 3, 4, take the geometric mean of the four, and wrap it in the brevity penalty.

The smoothing is what Lin and Och called method 1: add one to both numerator and denominator of every n-gram precision before taking the log. That avoids log 0 when a reference has no matching 4-gram, and the result stays close to the unsmoothed value on long candidates.

ROUGE-L scores the longest common subsequence (LCS) of the candidate and reference token sequences. An LCS keeps word order without requiring the matched words to sit next to each other, which is why ROUGE-L is the default summarisation metric. Compute the LCS length with a standard dynamic-programming table, then take recall as lcs / reference length and precision as lcs / candidate length. Combine them with F-beta, where beta = 1 gives the symmetric F1 form.

def lcs_length(a, b):
    n, m = len(a), len(b)
    if n == 0 or m == 0:
        return 0
    dp = np.zeros((n + 1, m + 1), dtype=np.int32)
    for i in range(n):
        for j in range(m):
            if a[i] == b[j]:
                dp[i+1, j+1] = dp[i, j] + 1
            else:
                dp[i+1, j+1] = max(dp[i+1, j], dp[i, j+1])
    return int(dp[n, m])

The numpy table keeps the recurrence easy to read, and pure Python lists would work too. A task that opts into ROUGE-L pays O(n m) per prediction-target pair, so cost grows with the product of the two lengths.

Accuracy is the fraction of predictions that match their target. In main.py it is a thin wrapper: accuracy(prediction, targets) calls exact_match on the whole target list, so it inherits the same rule (stripped strings, no other normalisation, any target may match). It gets its own function so the dispatcher can branch on metric_name without string comparisons inside the runner.

The single entry point is score(metric_name, prediction, targets), and it returns a float in [0, 1]. The runner doesn't branch on the metric name. It hands the call off and writes down the result. Lesson 75 glues this surface to the task spec from lesson 70.

def score(metric_name, pred, targets):
    if not targets:
        return 0.0
    if metric_name == "exact_match":
        return exact_match(pred, targets)
    if metric_name == "f1":
        return max(f1_score(pred, t) for t in targets)
    if metric_name == "bleu_4":
        return max(bleu4(pred, t) for t in targets)
    if metric_name == "rouge_l":
        return max(rouge_l(pred, t) for t in targets)
    if metric_name == "accuracy":
        return accuracy(pred, targets)
    raise ValueError(f"unknown metric_name: {metric_name}")

The code_exec metric is handled in lesson 72 and slots into the dispatcher there.

Here is what stays out: no model calls, no normalisation of generations beyond the post-process rules from lesson 70, no confidence intervals, and no BLEURT or BERTScore (those need a model and belong in a different lesson). What's left is the floor: five metrics, one tokenizer, one dispatcher.

main.py defines each metric as a free function, plus the dispatcher. A METRIC_TABLE dict lists how each metric aggregates (scalar or per-target max), corpus_mean averages per-task scores, and corpus_bleu is the pooled variant. The reference vectors live in the _reference_examples() function near the bottom of the file. The demo runs the dispatcher against eight examples, prints per-metric scores, and finishes with a corpus_bleu run over two pairs. The tests in tests/test_metrics.py pin the reference vectors and cover the edge cases: empty prediction, empty reference, empty target list, no shared tokens, exact match, and repeated-phrase clipping.

Read main.py top to bottom, since the functions run from simple to complex. exact_match is four lines and accuracy is a one-line wrapper around it. f1_score is about fifteen lines. BLEU and ROUGE-L are the heavy parts. The file has no comments, so the smoothing rule lives in bleu4 (add one to numerator and denominator) and the LCS recurrence in lcs_length. score is a plain if-chain over metric_name, not a lookup table.

Classical metrics are necessary but not sufficient. They reward surface overlap and miss meaning. Once you trust the classical floor, layer model-based metrics (BLEURT, BERTScore, GEval) on top; that's a later lesson. For now, get these five working and pin them with tests. You'll have a metric stack you can audit, run fast, and reproduce.