Capstone Lesson 27: Eval Harness with Fixture Tasks

An eval harness is a program that runs a fixed set of tasks through an agent and scores every result with a deterministic check. Without one, agent benchmarks tend to fail in three ways.

The first is unverified pass. The agent says it fixed the bug, a human glances at the diff, the suite is marked green, and three weeks later the regression test surfaces the same bug. The agent reasoned plausibly and fixed nothing.

The second is undetected regression. A change to the prompt template makes the agent 4% better on the loud task and 14% worse on the quiet one. With no goldset and no per-task score, the regression rides into main and surfaces only when a customer complains.

The third is per-task drift. You ran the eval on Monday with 100 tasks and on Friday with 95, because somebody renamed five fixtures. The pass rate looks like a 5% improvement. It isn't.

The harness turns each of these into a fact. It runs every fixture, every time, in a reproducible order, against a verifier that returns true or false on a deterministic check.

A FixtureTask is a directory. Its task.json declares an id, a goal (the prompt fed to the agent), and a verifier block that names a function in the harness's verifier registry and supplies its args. The setup is the buggy/ directory next to it, whose files get copied into the scratch dir. Tasks that use file_equals also have a sibling expected/ directory.

Three verifier shapes cover most useful tasks.

file_equals compares a named file against expected content after the agent runs. Use it for "fix this bug in this exact way" tasks.

regex_match matches the named file's contents against a regex. Use it for "the function must exist and return X" tasks, where many solutions are acceptable.

shell_exit_zero runs a command in the scratch dir with a timeout (timeout_seconds, default 10) and passes the task only if the command exits zero. The demo uses a plain subprocess.run; production wiring goes through the sandbox from lesson 26. Use it for "the tests must pass" tasks.

The harness runs each task k times. Pass@k is 1 - (1 - p)^k, where p is the empirical pass rate. The harness also reports raw counts so you can spot variance. Latency is wall-clock per sample, reported as a mean and a p95. Cost is whatever the candidate reports in cost_units (tokens, USD, or any unit you pick), reported as mean_cost per task and a total_cost across the run.

The candidate is a callable: Callable[[FixtureTask, str], SampleResult]. The harness creates the scratch directory with tempfile.mkdtemp() and passes its path as a plain string. How the candidate works is none of the harness's business. It could be a deterministic patch applier (handy for harness self-tests), a real LLM agent, or a fuzzer. The contract is the SampleResult.

main.py ships:

1. FixtureTask dataclass.
2. SampleResult dataclass: task_id, sample_index, latency_ms, cost_units, notes.
3. TaskReport, EvalReport dataclasses with to_dict().
4. VERIFIER_REGISTRY, a plain dict mapping verifier name to function. Built-in verifiers: file_equals, regex_match, shell_exit_zero.
5. EvalHarness class. Runs a directory of tasks against a candidate. Returns EvalReport.
6. Five fixture tasks bundled in tasks/:
   • off-by-one in fizzbuzz
   • missing return in factorial
   • typo in error message
   • empty function body
   • off-by-one in linked-list traversal
7. A deterministic reference candidate (apply_known_fixes) the harness uses to demonstrate a clean pass@1 of 1.0.
8. A demo that runs the reference candidate at k=1 and a noop candidate at k=3, and prints the results.

The fixture tasks are bundled as JSON files in tasks/, with starting files in tasks/<id>/buggy/ and the fixed versions in tasks/<id>/expected/. The harness copies buggy into a scratch dir, hands it to the candidate, and verifies against expected.

Pass@k is the probability of at least one pass in k independent samples. Real LLM agents are stochastic, so a single sample hides what the agent can do. A pass@1 of 0.6 looks like a failure. A pass@5 of 0.95 says the agent usually has the right answer and picks wrong on early samples. That points at sampling and ranking as the fix, not always more training. Pass@k makes the difference visible.

The harness reports pass@1 alongside pass@k because pass@k can paper over a real failure. If the model gets the right answer once in twenty tries, you do not have a useful agent.

Lesson 25 produced the gate chain. Lesson 26 produced the sandbox, which production wiring of shell_exit_zero should go through. Lesson 28 wraps each harness run in an OTel trace. Lesson 29 runs the end-to-end demo against one of the bundled fixtures and asserts pass@1 = 1.0 for the reference candidate.

cd phases/19-capstone-projects/27-eval-harness-fixture-tasks
python3 code/main.py
python3 -m pytest code/tests/ -v

The demo runs the reference candidate at k=1 and prints the EvalReport as JSON, including pass@1, mean latency, and a per-task breakdown. It then runs a noop candidate at k=3 and prints its pass@1 and pass@k. It exits nonzero if the reference candidate misses pass@1 of 1.0 or the noop candidate passes anything, and zero otherwise. The tests cover the verifier functions, the pass@k math, fixture loading, and the harness end-to-end against the bundled reference candidate.