Leaderboard Aggregation

The aggregator takes a list of EvalRun records, one per model and task, each carrying a score:

@dataclass
class EvalRun:
    model_id: str
    task_id: str
    metric_name: str
    score: float          # in [0, 1]
    category: str = "general"
    weight: float = 1.0

The runner in lesson 75 emits one record per (model, task) pair. The aggregator doesn't care how a score was produced, but it expects normalisation to have happened already: each score is in [0, 1]. category defaults to "general", and weight defaults to 1.0 but is ignored (see "What this lesson does not do").

A leaderboard row is a per-model summary record. Three tables come out:

Each row contains model_id, mean_score, mean_ci_lo, mean_ci_hi, win_rate, tasks_completed, and an optional categories map for per-category mean. pairwise_diffs produces a separate table of model-versus-model differences, and render_json serialises the rows for the JSON report.

Normalisation means putting every task's score on one common scale so no single metric controls the average. If one task scores in [0, 1] and another in [0, 100], the second quietly dominates the mean. The aggregator checks that each input score sits in [0, 1] and refuses the run otherwise. It also raises if the same (model, task) pair appears twice, since a duplicate would silently double-count a task. The fix belongs upstream: the metric should already return a fraction. Lessons 71 to 73 enforce that contract.

Mean score and win-rate answer different questions, so the harness reports both.

Mean score is the average over all of a model's runs. It's the headline number leaderboards report, and it's sensitive to outliers and to task imbalance.

Win-rate is the share of tasks where a model scores highest. On each task, every model whose score is within 1e-12 of the top score (checked with math.isclose) wins. Ties are not split: each tied model gets a full win, so the win-rates on a task can add up to more than 1. Win-rate equals wins divided by the number of tasks where the model has a score. It resists outliers and scale differences, at the cost of throwing away how large each margin was.

def _win_rate(model_id, runs_by_task):
    wins, total = 0, 0
    for task_id, runs in runs_by_task.items():
        scores = {r.model_id: r.score for r in runs}
        if model_id not in scores:
            continue
        total += 1
        best = max(scores.values())
        if math.isclose(scores[model_id], best, abs_tol=1e-12):
            wins += 1
    if total == 0:
        return (0.0, 0)
    return (wins / total, total)

It returns the rate and the task count, and the count becomes tasks_completed in the row.

The runner in lesson 75 ranks by mean by default. The win-rate column is in the same table for anyone who prefers that view.

A bootstrap confidence interval is a range for a statistic, built by resampling the observed data with replacement and recomputing the statistic many times. Here the statistic is a model's mean score. bootstrap_mean_ci draws indices into that model's own array of scores with replacement, computes the mean of each draw, repeats B times, and takes the percentile interval at level alpha. Each model is resampled on its own, so the per-model intervals ignore which tasks the models share.

For pairwise comparisons, pairwise_diffs first lines the two models up on the tasks both ran. bootstrap_pairwise_diff then bootstraps the per-task difference score_A - score_B the same way and reports the percentile interval. Read off whether the interval excludes zero. If it does, the difference is significant at level alpha. If it doesn't, significant is False and you should treat the two models as tied. That flag never changes the ranking: rows still sort by mean.

The low-level helpers (bootstrap_mean_ci, bootstrap_pairwise_diff) default to B=1000. The public aggregators (aggregate, pairwise_diffs) default to b=500 so the demo and tests stay quick. The default alpha is 0.05, and the bootstrap is plain numpy with no scipy.

A category mean is a model's average score within one group of tasks. Every EvalRun carries a category (default general), and the aggregator reports each one in the row's categories map. It's the math, reasoning, code, safety column on a typical leaderboard, and it shows when a model is good overall but weak at code, which the headline mean hides.

The leaderboard renders as a markdown table:

| Rank | Model | Mean | 95% CI | Win rate | Tasks |
|------|-------|------|--------|----------|-------|
| 1    | gpt   | 0.78 | 0.74-0.82 | 0.62 | 50 |
| 2    | claude| 0.75 | 0.71-0.79 | 0.34 | 50 |
| 3    | random| 0.10 | 0.07-0.13 | 0.04 | 50 |

Rows are sorted by mean score, the CI is rendered to two decimals, and long model ids are truncated to twenty characters. The header hardcodes 95% CI, so if you pass a different alpha the label will be wrong unless you edit it.

The scope is narrow. The aggregator doesn't run models and doesn't call the metric layer. It doesn't implement adaptive ECE or other calibration variants (those are lesson 73), and it doesn't implement task weighting. Tasks count equally here. Production leaderboards weight tasks, so the weight field (default 1.0) is left as an open hook that the aggregator ignores. Add weighting in a follow-up if you need it.

main.py defines EvalRun, LeaderboardRow, PairwiseDiff, aggregate, pairwise_diffs, bootstrap_mean_ci, bootstrap_pairwise_diff, render_markdown, and render_json. The demo builds a synthetic suite of three models and twelve tasks, aggregates, and prints the leaderboard plus the pairwise diff table. The tests in code/tests/test_leaderboard.py pin the bootstrap, the markdown rendering, the win-rate edge cases, and the empty-input behaviour.

Read main.py top to bottom: the data shapes (EvalRun, LeaderboardRow) first, then the aggregator, the bootstrap, and the rendering. Each function has a focused contract.

A natural next step is a hierarchical bootstrap that respects task families. Math problems are not independent of each other: an arithmetic error pattern can affect ten of them. That's a follow-up. The paired bootstrap on task-by-task differences, which the pairwise comparisons here already use, covers the case where models A and B ran the same hundred tasks. The goal is to get the floor right so the eval reports a number you can defend.