Language Model Evaluation Harness

An eval harness is a small program that runs a model against a fixed set of tasks, scores each answer with a fixed metric, and writes the scores to a JSON file you can diff. It has four parts: the task definition, the metric, the runner, and the leaderboard.

A new language model lands every week, and the claim is always that it does well. Well at what? The honest answer is the leaderboard you wrote yourself, because the vendor's leaderboard is the one they tuned to.

Without a harness in your repo you compare two models by vibes. With one, you compare them by score on a fixed task set with a fixed metric, in an output you can diff. The harness is the contract between yesterday's run and today's run, and without it regressions ship.

Over-fitting the harness to a single model is the trap, and the fix is to keep the harness small. You can read it in fifteen minutes. The tasks are small enough to ship in the repo. The metrics are written from scratch so a colleague can audit them, and model-specific code lives in the adapter. Swap the adapter and the leaderboard moves; swap the tasks and the leaderboard moves. Nothing else should move.

Task spec

Every example is one JSONL line:

{"id": "arith-00", "prompt": "compute: 2 + 2", "targets": ["4"], "metric": "exact_match"}

For metrics that need scoring helpers, extras carries the side payload:

{
  "id": "code-00",
  "prompt": "python: write a function f that doubles its input",
  "targets": ["ok"],
  "metric": "code_exec",
  "extras": {"io_pairs": [[1, 2], [3, 6]]}
}

A task is a .jsonl file under outputs/tasks/. The file name is the task name. All examples in a file share a metric.

The five fixture tasks

The metric contract

Every metric maps a prediction and its targets to a float in [0.0, 1.0]. Only code_exec reads extras; METRIC_FNS wraps the other four in three-argument lambdas so the runner can call every metric as (prediction, targets, extras). The harness averages the per-example scores to get a task score, then averages task scores to get the overall. The metric functions are tiny:

• exact_match: lowercase, collapse whitespace, equality.
• substring_contains: same normalization, substring test.
• multiple_choice: first character uppercased.
• rouge_l: LCS length divided by lengths of prediction and reference, F1 of precision and recall. With several targets it keeps the best F1.
• code_exec: execute the prediction in a restricted namespace, call f(x) on every pair in extras.io_pairs, and score the fraction that match. It ignores targets.

The code_exec metric runs the prediction in a stripped builtins namespace. The test asserts that a prediction containing import os scores 0.0: the stripped __builtins__ has no __import__, so exec raises, the metric catches the exception and returns 0.0. That keeps a code prediction away from the filesystem.

The model adapter

class ModelAdapter(Protocol):
    def generate(self, prompts: Sequence[str]) -> List[str]: ...
    @property
    def name(self) -> str: ...

The adapter is the seam between the harness and the model. This build ships ToyAdapter, a deterministic pattern matcher that returns the right answer for every prompt in the five fixture tasks. A real adapter calls the model and returns its output. The harness treats both the same.

The runner

run_task batches batch_size prompts at a time and dispatches to the metric function. run_leaderboard walks every task and averages. write_leaderboard emits JSON with a schema string, so a future format change does not silently break dashboards.

code/main.py is the runnable artifact. It is built in five steps.

Step 1: seed fixture tasks

seed_fixture_tasks(target_dir) writes the five .jsonl files. The first run of main.py seeds them when the directory is empty.

Step 2: load tasks

load_all_tasks(task_dir) reads every .jsonl and returns a dict from task name to a list of Example records. Comment lines starting with # and blank lines are skipped so contributors can annotate the files.

Step 3: implement metrics

Each metric is a small function with a unit test. The test suite includes 13 cases covering normalization, partial overlap, code execution, and unsafe code rejection.

Step 4: write the runner

run_task iterates batches and produces a TaskResult with score, correct count, total count, and latency. run_leaderboard walks all tasks and produces a Leaderboard with the overall average.

Step 5: emit JSON

write_leaderboard serializes the board. The --include-per-example flag dumps the per-example records so you can diff predictions against the previous run when scores move.

Run it:

python3 code/main.py

On first run the script seeds the fixtures, scores them with the toy adapter (which gets every fixture right), and writes outputs/leaderboard.json. The overall score is 1.0 with the toy adapter. The stub adapter test in test_main.py shows the same harness producing 0.0 when the adapter cannot answer.

Plugging in a real model means writing an adapter, a class with a name and a generate method. The shape:

class HttpAdapter:
    name = "vendor.v1"
 
    def __init__(self, endpoint, api_key):
        self.endpoint = endpoint
        self.api_key = api_key
 
    def generate(self, prompts):
        out = []
        for prompt in prompts:
            response = http_post(self.endpoint, prompt, self.api_key)
            out.append(response["text"])
        return out

Swap ToyAdapter for HttpAdapter at the top of main(). The harness, the tasks, the metrics, and the leaderboard stay the same.

Three habits to build in when you ship the harness in a real project:

• Pin the task files. The leaderboard.json this build writes carries neither task hashes nor copies of the JSONLs, so add one of them (Exercise 5 does the hash). Otherwise the score moves when the task file does, and you cannot tell which.
• Diff predictions, not just scores. The --include-per-example flag shows you what the model said the day the score dropped.
• Cap the batch size. Real adapters have rate limits, and a small batch size keeps the harness compatible across vendors.

outputs/skill-lm-eval-harness.md carries the recipe: JSONL task spec, five metrics, swappable adapter, batched runner, and a leaderboard JSON with a schema string. The task files in outputs/tasks/ are the fixtures. Copy them into a real project as starters.

1. Add a sixth task with a custom metric you write from scratch (BLEU-like overlap, BLEURT-like reference scoring, anything with a clear contract).
2. Extend code_exec to capture stdout and accept a list of expected stdouts as targets.
3. Add a leaderboard diff command: given two leaderboard.json files, print which tasks moved and by how much.
4. Cap latency per example. Wrap the adapter call in a timeout; surface a separate timeouts column in the leaderboard.
5. Pin task content with a sha256 in the leaderboard so a future reader can verify they scored the same tasks.

• The original lm-evaluation-harness for the production reference, much larger but the same shape.
• HuggingFace's lighteval for an alternative implementation of the same contract.
• Phase 19 lesson 46 covers the gradient accumulation patterns used in the training stack the harness scores.
• Phase 19 lesson 47 covers the checkpoint format you score against; pin the checkpoint hash in the leaderboard.
• Phase 19 lesson 48 covers the distributed training stack that produced the model under test.