End-to-End Research Demo
A demo is where every contract you wrote earlier has to compose. If one of them leaks, the demo is the lesson that catches it.
Learning Objectives
- Wire the auto-research loop end to end: hypothesis seed, experiment runner, scheduler, critic loop, paper writer.
- Compose the primitives from the four earlier Track D lessons through plain Python imports, not a framework.
- Run the loop to a self-terminating end and emit a single demo report that lists every stage's output.
- Keep the demo deterministic so the test suite can assert the final shape.
- Surface a clear failure mode when any stage's contract breaks, so the next stage does not run with a broken input.
What composes here
The demo runs five stages in order. The seed is a list of three hypotheses. The scheduler runs six experiments across them with three parallel slots, and the scheduler report lists one or more paper triggers. In the code, the runner's base rewards are 0.82, 0.55 and 0.15 for alpha, beta and gamma, the seed is 11 and the paper threshold is 0.7, so alpha is the branch that fires a trigger. The picker selects the single best result. The critic loop iterates on a draft built from that result, and the paper writer emits the final LaTeX, BibTeX, and manifest.
Why import, not copy
Each earlier lesson ships a main.py with public dataclasses and functions, and the demo calls a small _add() helper three times to put each lesson's folder on sys.path, then loads each main.py with importlib. A small _locate() helper looks for <lesson>/code/main.py first and falls back to <lesson>/main.py, so the demo runs from either layout. That is plain Python rather than framework wiring.
The inline stub stands in for lessons fifty through fifty-three: a small generator of seed hypotheses and a synchronous reward function. To use the real primitives from those lessons, point the demo at those lessons instead.
Determinism guarantees
The demo is deterministic by construction. The experiment runner is seeded numpy. The critic loop's reviser walks fixed dimensions in fixed order. The paper writer's prose generator is the mocked one from lesson fifty-four. The scheduler's UCB picker breaks ties on iteration order, not random choice.
Given the same seed, the demo emits the same report. The test checks this by running the demo twice and comparing the manifest.
The demo report shape
Each field comes verbatim from the upstream stage. The demo transforms no output; it composes them. That is the test the demo is.
Failure mode handling
Each stage either succeeds or raises a typed error.
Scheduler ........ returns SchedulerReport with stop_reason
in {queue_empty, max_experiments, deadline}
Best-result pick . raises NoTriggerError if no paper trigger fired
Critic loop ...... returns LoopResult with status converged or stopped
Paper writer ..... raises PaperValidationError on contract breakA failure in any stage short-circuits the demo with a typed exception. The tests pin the picker's half of this contract. test_no_trigger_raises asserts NoTriggerError when no branch fired a trigger. test_orphan_trigger_raises_best_result_error asserts BestResultError for a trigger that has no matching branch in the report. Both errors come from the picker, so the writer is never reached.
The best-result picker
The best-result picker is a small pure function that decides which branch becomes the paper. The scheduler emits paper triggers per branch, and the picker selects the branch with the highest mean reward across all triggers. Ties break alphabetically by branch id, so the demo is deterministic. The test pins the picker on a fixed scheduler report.
Wiring the critic loop
The critic loop in lesson fifty-five operates on a MiniPaper. The demo builds one from the picked branch: it populates the abstract with the branch id, seeds two sections (Introduction and Results), and sets originality_tag from the branch's mean reward (high if >= 0.8, medium if >= 0.6, low otherwise).
The reviser then iterates the draft to convergence, and the output goes into the paper writer.
Wiring the paper writer
The paper writer in lesson fifty-four operates on the full Paper shape, with figures and a bibliography. The demo upgrades the converged MiniPaper via mini_to_full_paper, which attaches one figure for the selected branch and a small synthetic bibliography built from the union of cite keys the critic suggested. Every cite the demo adds also goes into the bibliography list, so validation passes. If the critic suggested no cites at all, the demo adds a single <branch>-baseline entry and cites it from the Introduction, so the bibliography is never empty.
I'd keep the report a plain composition of upstream outputs, and I'd hold that line, because the moment the demo transforms something it stops being a test of the composition.
I'd also keep every stage raising a typed error rather than returning a bad value, so a failure stops the demo before the writer runs.
How to read the code
code/main.py defines BestResultError, NoTriggerError, DemoReport, pick_best_branch, build_mini_paper, mini_to_full_paper, and run_demo. The top of the file calls _add() three times to put the paper-writer, critic-loop, and scheduler lesson folders on sys.path, then loads each module with importlib and pulls out PaperWriter, CriticLoop, and IterationScheduler.
code/tests/test_e2e.py covers: demo runs end to end and emits a report with all six report fields populated, determinism across two runs, NoTriggerError when no branch crosses the threshold, PaperValidationError when the writer's contract breaks, paper manifest contains the picked branch's figure, and the scheduler stop reason is one of the expected values.
Going further
Three extensions are worth wiring once the demo is green. First, persistent state: each stage's result writes to a small JSON store, so a restart can resume without re-running the cheap stages. Second, a dashboard: the trace events from the scheduler and critic loop render as a single timeline. Third, real model calls: swap the mocked prose generator and the deterministic critic for model-driven ones, and the wiring does not change.
The demo's job is to prove that composition is the architecture. Three imported modules, one report. The next time you add a stage, the wiring grows by exactly one line.
Code
- main.py (python, 294 lines)
- tests/test_e2e.py (python, 131 lines)