Result Evaluator

A paired test compares two configurations on matched runs: the same seeds and the same data, run once with the candidate and once with the baseline. Each seed contributes one difference. The mean of those differences is the effect, and the standard error of those differences is the noise floor.

The reason to pair is that a single number from the runner cannot tell you whether a change is real. The same configuration with a different seed gives a different perplexity, so one gap between two runs might be noise.

I implement the test from scratch, with no scipy.stats. The math is small enough to read in one screen.

diffs    = [a_i - b_i for i in seeds]
mean     = sum(diffs) / n
variance = sum((d - mean) ** 2 for d in diffs) / (n - 1)
t_stat   = mean / sqrt(variance / n)
df       = n - 1
p_value  = two_sided_p(t_stat, df)

The two sided p value comes from a regularised incomplete beta function. I ship a small implementation built on the Lentz continued fraction. The whole thing is sixty lines of stdlib math.

Improvement is the signed fraction by which the candidate beats the baseline, and "beats" depends on the metric. Some metrics improve when they go up (accuracy, throughput). Others improve when they go down (loss, perplexity, wall time). Each metric carries a direction field that says which.

if direction == "higher_is_better":
    improvement = (candidate - baseline) / abs(baseline)
elif direction == "lower_is_better":
    improvement = (baseline - candidate) / abs(baseline)

The sign matters. A negative improvement on a higher is better metric means the candidate is worse, and the verdict path reads the sign and the magnitude together.

A flat threshold (improvement_threshold=0.02, two percent) decides whether the change is large enough to call. Below it the verdict is "noise" regardless of the p value. The loop has no use for a change the user could not measure.

The evaluator runs three independent computations and joins them in the verdict path. Each one is a pure function with no shared state.

Log normalisation takes the natural log of a metric before the improvement is computed, so metrics that grow exponentially can sit in one report next to linear ones. Perplexity is the standard case. It is exponential in loss, so a 0.1 drop in loss is a much larger drop in perplexity. Comparing perplexity directly across two configurations is fine. Blending it with linear metrics in a single report needs the transform.

The evaluator normalises any metric whose scale field is "log", and the threshold is then applied in log space. A perplexity drop from 32 to 28 has a raw log difference of log(28) - log(32) = -0.133. The evaluator then divides by the absolute baseline log and flips the sign for a lower is better metric, so improvement = (log(28) - log(32)) / abs(log(32)) with the sign flipped is about 0.0385. That is still above the two percent threshold.

if scale == "log":
    a = log(candidate)
    b = log(baseline)
else:
    a = candidate
    b = baseline

Metrics with scale="linear" (the default) skip the transform. One code path handles both.

One detail to keep straight: the improvement and the p value come from log-space values, but candidate_mean and baseline_mean in the Verdict are plain means of the original-scale metric (28 and 32 in the example, not their logs). Do not recompute the improvement from those two fields and expect it to match.

The runner from lesson fifty-two emits one final metrics blob per run. The paired test needs one blob per seed for the candidate and one per seed for the baseline. The orchestrator runs the same experiment under both configurations across a list of seeds and hands the evaluator two lists of ExperimentResult records.

The evaluator pairs them by seed, which lives in result.metrics["seed"], and walks the requested metric. Only seeds present in both lists are paired; seeds that appear in just one list are dropped silently. It raises a PairingError when the two lists share no seeds, when a result has no seed, or when a seed appears twice within one list, and the orchestrator should re-run.

A Verdict is the evaluator's output record for one hypothesis: the means, the signed improvement, the p value, both thresholds, a label, and a one line reason.

Verdict
  hypothesis_id          : int
  metric                 : str
  direction              : "higher_is_better" | "lower_is_better"
  scale                  : "linear" | "log"
  candidate_mean         : float
  baseline_mean          : float
  improvement            : float       (signed, fraction; see direction rules)
  p_value                : float | None  (None if n < 2)
  significance_threshold : float
  improvement_threshold  : float
  verdict                : "improved" | "regressed" | "noise" | "failed"
  rationale              : str

The label comes from a small decision table:

1. If any candidate or baseline result has terminal != "ok": verdict = "failed"
2. else if |improvement| < improvement_threshold:  verdict = "noise"
3. else if p_value is None or p_value > significance: verdict = "noise"
4. else if improvement > 0:                          verdict = "improved"
5. else:                                             verdict = "regressed"

The rationale is a one line, human readable sentence the orchestrator can log against the hypothesis id.

code/main.py defines MetricSpec, Verdict, Evaluator, the t statistic and incomplete beta helpers, and a deterministic demo. The t test itself is pure stdlib math. numpy only reads the metrics list and computes means and variances.

code/tests/test_evaluator.py covers the improved path, the regressed path, the noise path (small improvement), the noise path (low n), the failed terminal path, the log normalised path, the t test against a known reference value, and the pairing error.

Lesson fifty produced the hypothesis queue. Lesson fifty-one filtered out anything the literature had already settled. Lesson fifty-two ran the experiment under candidate and baseline configurations across seeds. Lesson fifty-three reads those runs and writes the verdict. The orchestrator stitches the four together:

for hypothesis in queue:
    literature = retrieval.search(hypothesis.text)
    if literature_settles(hypothesis, literature):
        attach(hypothesis, verdict="settled")
        continue
    candidates = runner.run_all(specs_for(hypothesis))
    baselines  = runner.run_all(baseline_specs_for(hypothesis))
    metric_spec = MetricSpec("perplexity", direction=LOWER, scale=LOG)
    verdict = evaluator.evaluate(hypothesis.id, metric_spec, candidates, baselines)
    attach(hypothesis, verdict)

That orchestrator is not part of this lesson. The four lessons compose into it with no glue beyond the dataclasses each one defines.