TAI BUI
Back to curriculum
CapstonePython

End-to-End Research Demo

A demo is where every contract you wrote earlier has to compose. If one of them leaks, the demo is the lesson that catches it.

TIME: ~90 minutesPREREQUISITES: Phase 19 lessons 50-56

Learning Objectives

  • Wire the auto-research loop end to end: hypothesis seed, experiment runner, scheduler, critic loop, paper writer.
  • Compose the primitives from the four earlier Track D lessons through plain Python imports, not a framework.
  • Run the loop to a self-terminating end and emit a single demo report that lists every stage's output.
  • Keep the demo deterministic so the test suite can assert the final shape.
  • Surface a clear failure mode when any stage's contract breaks, so the next stage does not run with a broken input.

What composes here

Mermaid diagram

The demo runs five stages in order. The seed is a list of three hypotheses. The scheduler runs six experiments across them with three parallel slots, and the scheduler report lists one or more paper triggers. In the code, the runner's base rewards are 0.82, 0.55 and 0.15 for alpha, beta and gamma, the seed is 11 and the paper threshold is 0.7, so alpha is the branch that fires a trigger. The picker selects the single best result. The critic loop iterates on a draft built from that result, and the paper writer emits the final LaTeX, BibTeX, and manifest.

Check: What composes here0/3
1. In the demo, what is the seed?Define
2. In what order does the demo run its five stages?Define
3. What is a paper trigger in the scheduler report?Define

Why import, not copy

Each earlier lesson ships a main.py with public dataclasses and functions, and the demo calls a small _add() helper three times to put each lesson's folder on sys.path, then loads each main.py with importlib. A small _locate() helper looks for <lesson>/code/main.py first and falls back to <lesson>/main.py, so the demo runs from either layout. That is plain Python rather than framework wiring.

Mermaid diagram

The inline stub stands in for lessons fifty through fifty-three: a small generator of seed hypotheses and a synchronous reward function. To use the real primitives from those lessons, point the demo at those lessons instead.

Check: Why import, not copy0/1
1. How does the demo load the paper-writer, critic-loop, and scheduler modules?Define

Determinism guarantees

The demo is deterministic by construction. The experiment runner is seeded numpy. The critic loop's reviser walks fixed dimensions in fixed order. The paper writer's prose generator is the mocked one from lesson fifty-four. The scheduler's UCB picker breaks ties on iteration order, not random choice.

Given the same seed, the demo emits the same report. The test checks this by running the demo twice and comparing the manifest.

Check: Determinism guarantees0/2
1. Why does the determinism test run the demo twice with the same seed?Explain
2. A developer changes the picker to break ties with a random choice. Two same-seed runs now sometimes pick different branches. Which guarantee is lost?Analyze

The demo report shape

Mermaid diagram

Each field comes verbatim from the upstream stage. The demo transforms no output; it composes them. That is the test the demo is.

Check: The demo report shape0/3
1. Which fields does DemoReport hold after a successful run?Define
2. Why does the demo transform none of the upstream outputs when it builds the report?Explain
3. A teammate rounds best_reward and rewrites the critic status into friendlier text before storing them in DemoReport. Which design principle does this violate?Analyze

Failure mode handling

Each stage either succeeds or raises a typed error.

text
Scheduler ........ returns SchedulerReport with stop_reason
                   in {queue_empty, max_experiments, deadline}
Best-result pick . raises NoTriggerError if no paper trigger fired
Critic loop ...... returns LoopResult with status converged or stopped
Paper writer ..... raises PaperValidationError on contract break

A failure in any stage short-circuits the demo with a typed exception. The tests pin the picker's half of this contract. test_no_trigger_raises asserts NoTriggerError when no branch fired a trigger. test_orphan_trigger_raises_best_result_error asserts BestResultError for a trigger that has no matching branch in the report. Both errors come from the picker, so the writer is never reached.

Check: Failure mode handling0/5
1. What is a typed error, as the demo uses the term?Define
2. What does the critic loop hand back to the demo when it finishes?Define
3. Why does each stage raise a typed error on a broken contract instead of returning a bad value?Explain
4. The scheduler report contains zero paper triggers. What does the demo do?Apply
5. run_demo builds the scheduler with max_experiments=6, and the run is cut off when it reaches that cap. Which stop_reason does the scheduler report carry?Apply

The best-result picker

The best-result picker is a small pure function that decides which branch becomes the paper. The scheduler emits paper triggers per branch, and the picker selects the branch with the highest mean reward across all triggers. Ties break alphabetically by branch id, so the demo is deterministic. The test pins the picker on a fixed scheduler report.

Check: The best-result picker0/4
1. What is the best-result picker?Define
2. How does the best-result picker choose between branches when more than one triggered a paper?Define
3. Branches alpha and beta both triggered with an identical mean reward of 0.81. Which becomes the paper?Apply
4. The demo runs with seed 11, base rewards alpha 0.82, beta 0.55, gamma 0.15, and a paper threshold of 0.7. Which branch does run_demo pick?Apply

Wiring the critic loop

The critic loop in lesson fifty-five operates on a MiniPaper. The demo builds one from the picked branch: it populates the abstract with the branch id, seeds two sections (Introduction and Results), and sets originality_tag from the branch's mean reward (high if >= 0.8, medium if >= 0.6, low otherwise).

The reviser then iterates the draft to convergence, and the output goes into the paper writer.

Check: Wiring the critic loop0/2
1. A picked branch has a mean reward of 0.65. Which originality_tag does build_mini_paper set?Apply
2. What originality_tag does build_mini_paper("gamma", 0.8) set?Apply

Wiring the paper writer

The paper writer in lesson fifty-four operates on the full Paper shape, with figures and a bibliography. The demo upgrades the converged MiniPaper via mini_to_full_paper, which attaches one figure for the selected branch and a small synthetic bibliography built from the union of cite keys the critic suggested. Every cite the demo adds also goes into the bibliography list, so validation passes. If the critic suggested no cites at all, the demo adds a single <branch>-baseline entry and cites it from the Introduction, so the bibliography is never empty.

Check: Wiring the paper writer0/5
1. What does mini_to_full_paper do when upgrading a converged MiniPaper?Define
2. Why does the demo need mini_to_full_paper between the critic loop and the paper writer?Explain
3. The critic loop suggested no citations. What does mini_to_full_paper do so validation still passes?Apply
4. A MiniPaper has intro cites [k1] and results cites [k1, k2]. What bibliography keys does mini_to_full_paper build?Apply
5. A change adds a citation key to the Introduction's cites but adds no matching bibliography entry. Given that the demo keeps every cite in the bibliography so validation passes, what should you expect?Analyze
Author's Note

I'd keep the report a plain composition of upstream outputs, and I'd hold that line, because the moment the demo transforms something it stops being a test of the composition.

I'd also keep every stage raising a typed error rather than returning a bad value, so a failure stops the demo before the writer runs.

How to read the code

code/main.py defines BestResultError, NoTriggerError, DemoReport, pick_best_branch, build_mini_paper, mini_to_full_paper, and run_demo. The top of the file calls _add() three times to put the paper-writer, critic-loop, and scheduler lesson folders on sys.path, then loads each module with importlib and pulls out PaperWriter, CriticLoop, and IterationScheduler.

code/tests/test_e2e.py covers: demo runs end to end and emits a report with all six report fields populated, determinism across two runs, NoTriggerError when no branch crosses the threshold, PaperValidationError when the writer's contract breaks, paper manifest contains the picked branch's figure, and the scheduler stop reason is one of the expected values.

Check: How to read the code0/2
1. A test builds a SchedulerReport with branches=[] and paper_triggers=["ghost"], then calls pick_best_branch. What happens?Apply
2. A test builds a full paper from build_mini_paper("alpha", 0.9), sets its title to an empty string, and calls writer.write. What happens?Apply

Going further

Three extensions are worth wiring once the demo is green. First, persistent state: each stage's result writes to a small JSON store, so a restart can resume without re-running the cheap stages. Second, a dashboard: the trace events from the scheduler and critic loop render as a single timeline. Third, real model calls: swap the mocked prose generator and the deterministic critic for model-driven ones, and the wiring does not change.

The demo's job is to prove that composition is the architecture. Three imported modules, one report. The next time you add a stage, the wiring grows by exactly one line.

Warm-up0/3
1. You wrote several modules that each pass their own tests, then chain them in one script. What does running the chain mainly test?Warm-up
2. A pipeline stage receives bad input from the stage before it. Which behavior makes the bug easiest to find?Warm-up
3. A unit test asserts that a function returns an exact value. What must be true of the function for that test to pass on every run?Warm-up
Check: Going further0/3
1. The demo imports each earlier lesson's module rather than using a framework. What does that buy when you add another stage?Explain
2. You add a literature-retrieval stage between the scheduler and the picker. Which design fits the demo's approach?Design
3. The writer crashes after the scheduler and critic loop already finished. You want a restart to skip those cheap stages. Which extension fits?Design

Code