Experiment Runner

A subprocess is a separate operating system process that the runner launches and watches. It has its own address space, and the parent holds a signal handle it can use to stop it.

A research loop runs untrusted code. The hypothesis came from a sampler and the experiment script came from the same path, so running either in-process risks a crash that takes the orchestrator down with it. A subprocess is the simplest isolation the language ships.

The runner here does not implement full sandboxing. There is no cgroup, no seccomp filter, no namespace remapping. What it has is a wall clock timeout, a polling loop for memory growth, and a kill path that terminates the process when either limit trips. That is the runtime contract every more elaborate sandbox extends, and I kept it small enough to read in one sitting.

ExperimentSpec
  spec_id        : str            (stable id, "exp_001")
  hypothesis_id  : int            (link back to the queue from lesson 50)
  script_path    : str            (path to the python script to run)
  config         : dict           (written to a temp config.json; its path is argv[1])
  seed           : int            (deterministic seed for the experiment)
  wall_timeout_s : float          (hard timeout, killed on exceed)
  memory_cap_mb  : int            (soft cap, polled; killed on exceed)
  metric_keys    : list[str]      (which fields the evaluator will read)

The script lives on disk, and the runner writes the config to a temp file and passes that path to the script as argv[1]. The script is expected to print a json line on stdout whose keys are a superset of metric_keys. It may print several such lines, and the last one is final. Anything else on stdout is captured, but the metrics parser ignores it.

The runner is one class with one main method. The poller is a small thread that wakes once per poll interval and reads the subprocess resident set size from /proc/<pid>/status, falling back to ps -o rss=. If both fail, the poller logs a warning (once per process) and exits.

A soft memory cap is a limit the runner enforces by polling the subprocess's resident set size, not through the operating system. Hard memory caps need resource.setrlimit and only work on POSIX, so I went with a portable approach: read the resident set size from the platform and kill the subprocess if it exceeds the cap.

The cap is soft because the poller has a non zero interval. A process can spike above the cap between polls and then drop back. The runner records the maximum observed RSS, so the evaluator can see how close the run came to the limit.

On systems where neither /proc nor ps can report RSS, the poller logs a one time warning (once per process) and exits. The wall clock timeout still applies.

The runner drains both pipes on completion. It scans stdout line by line and takes the last line that parses as json with all required metric_keys as the metrics blob. Earlier json lines that qualify are kept in the result as intermediate_metrics, which the evaluator can use for learning curves.

Stderr goes into the result verbatim. The runner does not raise on a non zero exit code. It records the code in the result instead. A non zero exit is labelled "crash" even when the script printed metrics, so the evaluator treats partial runs as failures by default. The exceptions are runs the runner itself killed: "oom" and "timeout" take precedence over "crash". A zero exit with no line covering metric_keys is also "crash".

An ablation table is the set of results from running one base spec many times while a single configuration knob takes a different value on each run.

def ablate(base: ExperimentSpec, knob: str, values: list[Any]) -> list[ExperimentSpec]:
    ...

Given a base spec and a knob name, the helper returns one spec per value with config[knob] overridden. Each spec gets a derived spec_id (f"{base.spec_id}_{knob}_{value}"). The runner ships an AblationRunner that runs them in order and returns an AblationTable whose rows are (value, ExperimentResult) pairs in sweep order.

Why one knob at a time? Full factorial sweeps blow up exponentially and produce results the evaluator cannot interpret. One knob gives a clean axis the evaluator can plot. Multi knob sweeps are supported only as repeated single knob ablations, composed by the caller.

Determinism here means the same spec produces the same metrics on every run. Each spec carries a seed, and the runner forwards it to the script through the config dict (config["__seed"] = spec.seed). The mock experiment scripts in code/experiments/ honour the seed and produce identical metrics across runs. The evaluator in lesson fifty-three depends on this. Without determinism, a "regression" might just be a different random initialisation.

The mock experiment is code/experiments/sparsity_experiment.py. It is a real script that reads its config file, simulates a small training run with a numpy random pass, and prints a json metrics blob. It honours a sleep_s knob for testing timeouts and an allocate_mb knob for testing the memory poller.

A second script, experiments/crashing_experiment.py, prints one json line that lacks the metric keys, writes to stderr, and exits with code 3 so the crash label has something to test against. The simulation in the first script trains nothing real. It is a numerical computation that mimics the shape of a training loop: a loss curve, a final perplexity, a wall time. The subject here is the runner, not the simulation. A real experiment script would import a model.

ExperimentResult
  spec_id              : str
  hypothesis_id        : int
  exit_code            : int
  terminal             : "ok" | "timeout" | "oom" | "crash"
  wall_time_s          : float
  peak_rss_mb          : float | None
  metrics              : dict
  intermediate_metrics : list[dict]
  stdout_tail          : str
  stderr_tail          : str

The evaluator reads metrics and terminal first. If terminal is anything other than "ok", the experiment counts as a failed run and the evaluator's verdict is automatic. Otherwise the metrics go through the significance test.

code/main.py defines ExperimentSpec, ExperimentResult, ExperimentRunner, AblationRunner, and a deterministic demo. Subprocess management lives in one class, the memory poller is a small thread, and the ablation helper is a single function.

code/experiments/sparsity_experiment.py is the mock experiment used in tests. It takes its config file path from argv, prints intermediate json lines per step, and ends with the final metrics line. code/experiments/crashing_experiment.py is the failing counterpart.

code/tests/test_runner.py covers the success path, the timeout path, the crash path, the ablation table, and the determinism check across two runs.

Lesson fifty generates the hypothesis. Lesson fifty-one filters out anything the literature already settled. Lesson fifty-two, this one, runs the experiment for whatever is left. Lesson fifty-three reads the result, runs the significance test, and writes the verdict the orchestrator stores against the hypothesis id.