Task Spec Format

A frozen spec is one task schema, one closed set of metric names, and a validator that rejects records outside them. The validator checks every record before it reaches the runner.

Without one, a research codebase picks up eval scripts faster than it picks up tests. Six months in, notebooks carry their own JSON shapes, metrics get reimplemented twice, and runs can't be compared. The fix is boring: pick a schema, write a validator, reject the rest.

The shape borrows ideas from BIG-bench, HELM, and lm-eval style harnesses, but the field names are ours. Each field has one owner. The runner reads the task, the metric reads the targets, and the post-process step normalises the generation. No field changes mid-pipeline.

A task record is one JSON object on one line. The harness reads tasks.jsonl and validates each line independently, so a bad line fails that record and leaves the run alone.

{
  "task_id": "arith_001",
  "category": "arithmetic",
  "prompt": "Compute the result. Question: 17 + 24\nAnswer:",
  "targets": ["41"],
  "metric_name": "exact_match",
  "few_shot_examples": [
    {"prompt": "Question: 2 + 2\nAnswer:", "completion": "4"}
  ],
  "post_process": "strip_whitespace",
  "metadata": {"difficulty": "easy"}
}

Six fields are required: task_id, category, prompt, targets, metric_name, post_process. few_shot_examples and metadata are optional. A top-level field outside those eight fails validation.

task_id is a string made of letters, digits, underscores, and hyphens only (`^[A-Za-z0-9_-]+), unique across the file.

category is one of arithmetic, mcq, code_exec, classification, summary. It decides which metric and post-process pair is legal. The legal pairs are: arithmetic takes exact_match or f1; mcq takes exact_match or accuracy; code_exec takes only code_exec; classification takes exact_match, accuracy, or f1; summary takes exact_match, f1, bleu_4, or rouge_l. An mcq task also needs exactly one target, a single letter from A to E.

prompt is a non-empty string with no trailing whitespace. A prompt that looks like it carries its own few-shot block is rejected when few_shot_examples is empty. The check is a heuristic: it lowercases the prompt and flags it if more than one of question:, q:, or example: appears. Few-shot rendering happens in the runner, and the author supplies the examples in few_shot_examples.

targets is a non-empty list of strings. For exact_match, a match on any element counts. For f1 and rouge_l, the highest-scoring target wins. For mcq, the list holds exactly one element.

metric_name is one of exact_match, f1, bleu_4, rouge_l, accuracy, code_exec. The vocabulary is closed, so a new metric means a new lesson and a new entry here.

few_shot_examples is a list of {prompt, completion} pairs, capped at eight entries to keep prompts bounded.

post_process is one of none, strip_whitespace, lower, extract_letter, extract_code_block, extract_first_line. Each rule does one deterministic thing, and rules can't be combined.

The validator returns two lists. One holds the validated records. The other holds error records, each with the offending line, the violated rule, and the field at fault. Each record gets one error at most, for the first rule it breaks. A task id seen twice and a line that is not valid JSON are reported too. main.py stops at returning the two lists. Deciding what a non-empty error list means for a run is the runner's job in lesson 75, and the intended policy is that it refuses to start.

Few-shot rendering is the step that turns a task's examples and prompt into the text a model sees. The runner puts the examples in front of the prompt, separated by blank lines. One code path serves all models, so the model is the only source of variance. Authors write examples once, not once per provider.

def render(task):
    parts = []
    for ex in task.get("few_shot_examples", []):
        parts.append(ex["prompt"] + " " + ex["completion"])
    parts.append(task["prompt"])
    return "\n\n".join(parts)

Post-processing is the step between generation and metric that turns raw model output into something comparable to a target. It is deterministic and stateless.

• none returns the string unchanged.
• strip_whitespace strips leading and trailing whitespace.
• lower lowercases the string.
• extract_letter returns the first standalone letter from A to E (one not touching another letter), and only if there is none falls back to the first A-E character anywhere, or an empty string. It is used for MCQ. In Answer: C is correct it returns C, because the A in Answer is part of a word.
• extract_code_block returns the body of the first triple-backtick fenced block, used for code-exec. If the text has no fence it returns the whole text unchanged.
• extract_first_line returns the first non-empty line, used for summary classification.

A task that needs a rule outside this list belongs in a new lesson.

The spec doesn't score, call a model, or run code. Those come in lessons 71, 72, and 75. What it does is freeze the contract they all honour.

The 10-task fixture holds two arithmetic items, two MCQ items, two code-exec items, two classification items, and two summarisation items, and the validator passes all 10. A separate fixture (tasks_bad.jsonl) holds 10 records built to fail, and the validator returns exactly one error for each. They cover a missing field, a bad id, a bad category, an illegal pair, empty targets, trailing whitespace, an unknown field, a bad metric, a multi-target MCQ, and a bad post-process rule. The embedded few-shot, malformed JSON, and duplicate id checks are not in that file. The tests in tests/test_spec.py cover them.

main.py defines TaskSpec, validate_task, validate_file, and load_fixtures, which writes both fixture files into a directory you give it. There is no CLI beyond demo(), which runs when you execute the file. The render and post-process helpers sit next to validation, so the runner in lesson 75 can import a single module.

Read main.py top to bottom, then tests/test_spec.py. The tests pin each validation rule and each post-process behaviour. The demo at the bottom of main.py writes both fixtures to a temporary directory, validates them, and prints a summary.

Real eval suites grow categories the way schemas grow columns. The sober move is to refuse a new category until it arrives with a metric, a post-process rule, and at least one fixture task. Treat the spec like a database migration: review the change, version it, and ship tests with it. The validator is the gate.