End-to-End Eval Runner

The eval runner is the integration point: the one program that takes a task spec and a set of model adapters and produces a leaderboard with calibration attached. Lessons 70 through 74 each own one module, and the runner composes them. It imports their code instead of copying any of it.

A model adapter is the seam between the runner and any model, and it needs one method. The interface is small on purpose.

class ModelAdapter:
    model_id: str
 
    def generate(self, prompt: str, task: TaskSpec) -> Generation: ...

ModelAdapter is a plain class with a model_id and a generate method that subclasses override. Generation is a dataclass with:

• text: the model's free-form output
• confidence: a float in [0, 1] representing the model's self-reported probability for the answer (defaults to 0.5, and the runner clamps it into range)
• token_nll: sum of negative log-likelihoods over the generated tokens (defaults to 0.0)
• token_count: number of generated tokens (defaults to 0)

The runner never reads token_nll from the returned Generation. Each mock adapter instead appends (nll, token_count) pairs to its own token_stats list, and the perplexity block in the report is built from those. The runner ships three mock adapters. RuleBasedAdapter is deterministic and near-perfect. NoisyAdapter is overconfident and often wrong. BiasedAdapter is good at one category and terrible at another. The demo runs all three over the lesson 70 fixture.

The runner uses concurrent.futures.ThreadPoolExecutor to run work in parallel. The work list is every adapter crossed with every task, pooled into one executor rather than one pool per model. The worker count is min(max_workers, len(work)), where max_workers defaults to eight. The demo passes six, so its 3 adapters times 10 tasks give 30 work items and 6 workers. Threads are enough here because real model calls spend their time waiting on the network. The code-exec path spawns its own subprocess inside the task, so the executor only schedules the wait.

Tests need a fixed execution order, so the runner exposes run_eval(adapters, tasks, parallel=False) to pin it.

_score_one touches each (adapter, task) pair once, in this order:

1. Render the prompt (few-shot prefix plus the prompt body).
2. Call the adapter and time the call.
3. Post-process the generation per the task's rule.
4. Dispatch to the metric layer.
5. Derive the correct flag and build a TaskResult carrying the score and metric metadata.

Calibration bookkeeping happens later, in run_eval. Once every result is collected and sorted by (model_id, task_id), the runner appends one (confidence, 1.0 or 0.0, score) triple per result to that model's calibration buffer. Sorting first keeps the buffer order the same for parallel and sequential runs.

The correct signal is score >= 0.999999 for exact_match-style metrics (exact_match, accuracy, code_exec) and score >= threshold for graded metrics. The threshold lives in _correct_from_score as an argument defaulting to 0.5, and run_eval does not expose a way to override it.

Once all tasks have results, the runner calls aggregate and pairwise_diffs from lesson 74 and CalibrationReport.from_predictions from lesson 73. The output is a single report with these keys:

{
  "leaderboard": [...],
  "pairwise": [...],
  "calibration": {
    "model_id_a": {"ece": 0.04, "brier": 0.10, "populated_bins": 8, "n_samples": 10},
    ...
  },
  "perplexity": {
    "model_id_a": {...},
    ...
  },
  "summary": {
    "tasks": 10,
    "models": 3,
    "task_runs": 30,
    "wall_seconds": 1.2
  }
}

Perplexity per model comes from the adapters' token_stats, not from the Generation objects. EvalReport.to_json() serializes the envelope, though the demo never calls it. The demo prints a markdown table (ready to paste into a PR review), the summary as JSON, and one line per pairwise difference.

The demo runs three mock adapters over the ten fixture tasks from lesson 70. Wall time should sit under ten seconds, and a clean run exits with code zero.

A clean run means these held:

• The fixture tasks validated under lesson 70. A validation failure raises a RuntimeError before anything runs.
• The tasks scored under lessons 71 and 72 and the calibration report aggregated under lesson 73, with no exception escaping.
• The leaderboard is non-empty, rule_based ranks first, and rule_based is not last.

The demo checks only those leaderboard conditions. There is no random adapter. On a failed check it prints a line starting with ERROR: and exits with code 1 (empty leaderboard), 2 (rule_based not on top), or 3 (rule_based at the bottom). It does not emit a structured JSON error.

The runner does not call a real model, and it has no API key flow or rate-limit handling. It has no streaming or partial generation: the adapter returns one generation per call. It does no retries and no caching. Those concerns belong to the adapter layer. The runner itself is metric-agnostic and provider-agnostic.

main.py is the integration. It pulls in the other five lesson modules through a small _load_sibling helper that resolves them by relative path. It looks for each sibling at code/main.py first and falls back to main.py, so the same file works whether lessons 70 to 74 keep a code/ folder or sit flat. Generation, EvalReport, and ModelAdapter are defined locally, and the mock adapters sit near the bottom of the file.

Read it top to bottom. Skim the imports, then run_eval, then _score_one, then the adapters. The demo at the end is the entry point.

The tests in code/tests/test_runner.py pin the adapter interface, the single-pass loop, parallel-versus-sequential equivalence, the calibration buffer, and the JSON envelope shape.

This runner is the floor. A production eval system adds a results cache keyed by (task_id, model_id, model_version), a cost ledger that tracks dollars and tokens per run, a retry layer that backs off on rate limits, a sampling policy for pass-at-k tasks, and a streaming output format for long suites. Each is a single concern that wraps the runner without touching the metric or aggregation layers. That separation is the point of the contract.

Add an adapter for a real provider once the mocks work. Pick one with a free tier, write thirty lines of glue, and watch the leaderboard light up. Then add a second provider and let the harness do the work.