Perplexity and Calibration

Perplexity is the exponentiated average negative log-likelihood per token. Lower is better. A perplexity of one means the model assigns probability one to every actual token. A perplexity equal to the vocabulary size means the model is uniform and learned nothing. Real numbers fall in between: a strong 2026 base model on WikiText-103 sits around eight to twelve, and a bad one on the same text sits at fifty plus.

The harness does not compute log-probabilities itself, because those come from the model adapter. What the harness does is aggregate. It takes a list of per-sequence summed negative log-likelihoods and a matching list of token counts per sequence, and returns corpus perplexity.

The core arithmetic is below. In main.py, perplexity delegates to PerplexityResult.from_token_nll, which adds the validation.

def perplexity(neg_log_probs, token_counts):
    total_nll = sum(neg_log_probs)
    total_tokens = sum(token_counts)
    return math.exp(total_nll / total_tokens)

The implementation returns NaN when the total token count is zero, and from_token_nll raises a ValueError if any negative log-likelihood is negative (or if the two lists differ in length). The mistake I'd check for first is a forgotten negation. An adapter that returns log p instead of -log p produces a perplexity below one, which is impossible, so the function rejects it as a contract violation.

Expected calibration error (ECE) is the average gap between confidence and accuracy, taken over bins of predictions and weighted by bin size. It groups predictions by confidence into a fixed number of bins, then compares each bin's mean confidence to its observed accuracy.

The standard formulation uses ten equal-width bins on [0, 1]. The implementation accepts any positive integer count, and the bins parameter (default 10) lets the runner pick a different count.

ECE is biased by bin count and sample size. With ten bins and a hundred predictions, a 0.02 ECE is hard to tell apart from noise. So the implementation returns the number of populated bins alongside the ECE, so the runner can decline to report a single number on too few samples.

The Brier score is the mean squared error between predicted probability and the actual outcome, scored per prediction. For binary outcomes it is mean((p_i - y_i)^2).

ECE compares each bin's mean confidence to its accuracy. Gaps in different bins add up as absolute values, so they never cancel each other. Inside a single bin, though, overconfident and underconfident predictions average out, and the bin can look calibrated. Brier scores every prediction against its outcome, so that spread shows up directly in the number.

Brier decomposes into reliability, resolution, and uncertainty. main.py provides both the score and brier_decomposition, which returns a dict with reliability, resolution, uncertainty, and brier. The decomposition is a standalone function: CalibrationReport does not include it, and main.py has no runner, so whatever calls the harness decides whether to log it.

def brier_score(confidences, correct):
    conf = np.asarray(confidences, dtype=np.float64)
    corr = np.asarray(correct, dtype=np.float64)
    _validate_probs(conf, corr)
    if conf.size == 0:
        return 0.0
    return float(np.mean((conf - corr) ** 2))

A reliability diagram plots predicted confidence against empirical accuracy in each bin, and the diagonal marks perfect calibration. The function returns three arrays: per-bin average confidence, per-bin average accuracy, and per-bin count. The plotting code lives downstream, so the code here stops at the data shape.

That tuple is what a calling layer needs to draw the plot or to compute a custom ECE variant (adaptive ECE, sweep ECE, and so on). The arrays come back as numpy arrays so downstream code skips the conversion.

Confidence is any number in [0, 1] attached to a prediction, and the harness does not assume it comes from softmax. For multiple-choice tasks the natural choice is a softmax over option log-likelihoods. For free-text it is the model's self-reported probability or the exponential of the average log-likelihood. The eval consumes the number as given, and producing it is the adapter's job.

• All predictions wrong: ECE is the average confidence, Brier is high, and perplexity is whatever the model thinks of the text.
• All predictions correct with high confidence: ECE and Brier both sit near zero.
• A perfectly uncertain predictor at p=0.5: ECE is abs(0.5 - accuracy), and Brier is exactly 0.25.
• Empty input: ECE returns (0.0, 0), Brier returns 0.0, and reliability returns zero-filled arrays. Perplexity returns NaN for the zero-token case. None of these paths emit a warning. The runner inspects the values and decides whether to report or skip.

The tests pin these cases. A real model on a real benchmark is unlikely to hit them, but a buggy adapter or a tiny sample can, and the runner should survive that.

Calibration is a per-model report, where F1 is a per-task metric. The calling runner (which lives outside main.py) accumulates (confidence, correct) pairs across the whole eval and computes ECE, Brier, and reliability data once. Perplexity runs over a held-out text corpus, separate from the task-by-task scoring.

The interface is:

report = CalibrationReport.from_predictions(confidences, correct)
report.ece          # float
report.brier        # float
report.reliability  # tuple of three numpy arrays
report.populated_bins  # int

PerplexityResult.from_token_nll(neg_log_probs, token_counts) returns the perplexity, the average negative log-likelihood per token, and the total token count. CalibrationReport has no perplexity field, so the two results travel separately.

The code does not call a model and does not implement softmax. It does not estimate confidence from output tokens, since that is the adapter's job. Temperature scaling and Platt scaling are post-hoc fixes and belong in a different lesson. The goal here is narrower: make the three numbers (perplexity, ECE, Brier) trustworthy and reproducible.

main.py defines perplexity, expected_calibration_error, brier_score, brier_decomposition, reliability_diagram, and the CalibrationReport / PerplexityResult dataclasses. The demo runs on synthetic predictions where the ground truth is known: a well-calibrated model, an overconfident one, and an underconfident one. The tests in tests/test_calibration.py pin the edge cases plus reference values for the synthetic predictors.

Read main.py top to bottom. Functions run from scalar to vector to report. The functions carry no docstrings (only the module has one), so the math and the contract live in this lesson.

Calibration is one of the least-reported axes in published eval. Many leaderboards report a single accuracy number and stop there. A model that wins on accuracy but loses on Brier can be a worse production deployment than one that scores a few points lower on accuracy and reports its uncertainty reliably. With the calibration plumbing in place, add temperature scaling on a held-out validation slice, recompute ECE, and watch the gap shrink. That is a separate lesson, but the floor lives here.