CapstonePython (agent + sandbox), LaTeX (output)

Capstone 05 — Autonomous Research Agent (AI-Scientist Class)

Sakana's AI-Scientist-v2 published full papers, Agent Laboratory ran the experiments, and Allen AI shared traces. The 2026 shape is plan-execute-verify tree search over experiments, with budgeted cost, sandboxed code execution, a vision-feedback LaTeX writer, and an automated NeurIPS-style reviewer ensemble. The capstone: build one, run it end to end within $30 per paper, and survive the sandbox-escape red team Sakana documented.

TIME: 40 hoursPREREQUISITES: Phase 2 (ML), Phase 3 (deep learning), Phase 7 (transformers), Phase 10 (LLMs from scratch), Phase 14 (agents), Phase 15 (autonomous), Phase 16 (multi-agent), Phase 18 (safety)

Learning Objectives

Build a best-first tree search over experiment nodes, scored by novelty, quality, and remaining budget.
Run every LLM-generated experiment in a resource-capped sandbox with no network egress, and enforce a hard $30 per-paper budget.
Close the writer loop: compile LaTeX, critique the rendered PDF with a vision model, and gate the draft with a five-judge reviewer ensemble.
Red-team the sandbox against fork bombs, network exfiltration, and filesystem escapes, and ship a reproducibility bundle with every paper.

Problem

An autonomous research agent is a program that takes a seed idea and carries it through experiments, a written paper, and an automated review with no human steering each step. They crossed a threshold in 2026. Sakana AI's AI-Scientist-v2 was published in Nature with generated papers that cleared workshop peer review. ShinkaEvolve (ICLR 2026) extended the line to evolving hypotheses. AMD's Agent Laboratory shipped reproducible traces.

There is no magic in any of them. Each one is a plan-execute-verify loop running over a tree of candidate experiments, with cost caps, seed-bound sandboxes, and automated review. The craft is in the loop, the budget, and the safety story.

You learn the loop by implementing one against a seed idea in a narrow domain, for example attention-sparsity ablations on a 100M-parameter transformer. Don't expect to discover anything new on the first run. What you're building is the infrastructure: the tree search, the experiment sandbox, the writer-reviewer loop, and the red-team report. The Sakana team documented sandbox-escape failures, and your agent has to pass the same red team.

Warm-up0/3

1. An agent will run code that an LLM wrote, unattended. Which precaution matters most before it starts?Warm-up

2. In a plan-execute-verify agent loop, what is the verify step for?Warm-up

3. Why grade an output with several different LLM judges instead of one?Warm-up

Concept

The agent is a best-first tree search. A node is an experiment specification: (hypothesis, config, code, expected outcome). An expand step proposes children by making small edits, such as swapping the optimizer, shifting the batch size, or ablating a component. Each child runs in a fresh sandbox with a hard resource cap, and its results feed a scoring function that ranks nodes by a weighted blend of novelty, quality, and remaining budget. The tree grows until the budget runs out, then the best branch gets written up.

The writer is multimodal. It generates a LaTeX draft, compiles it, renders the figures, and feeds the rendered PDF back into Claude Opus 4.7's vision mode for critique of layout, figure legibility, and claim-evidence alignment. A reviewer ensemble of five LLM judges then emits NeurIPS-style scores (novelty, rigor, clarity, reproducibility, impact). If the average falls below threshold, the paper goes back to the writer with the critique attached.

Safety carries real weight here. Every experiment runs in an E2B or Daytona sandbox with no network egress, bounded wall-clock time, and pinned resource limits. The agent's code-generation step passes through a policy layer that blocks syscalls capable of escaping the sandbox. The red-team report reproduces the attack surface Sakana documented: fork bombs, filesystem escapes, and LLM-written network calls.

Check: Concept0/12

1. In this agent's tree, what is a node?Define

2. What is the tree search in an autonomous research agent?Define

3. What is the vision critique step in the writer loop?Define

4. What is the reviewer ensemble?Define

5. What does the novelty score measure?Define

6. Why does the scoring function include remaining budget alongside novelty and quality?Explain

7. Why does every experiment run with no network egress, even though the agent wrote the code itself?Explain

8. Why does the writer render the compiled PDF and critique the pages instead of checking only the LaTeX source?Explain

9. Why does a draft that scores below threshold go back to the writer with the critique attached?Explain

10. A coworker proposes expanding every node equally instead of using best-first search. What does the tree lose?Explain

11. After swapping the five judges for one, more known-bad papers get accepted. Which explanation fits the lesson's design?Analyze

12. Your run has spent $26 of the $30 ceiling, and the tree is still proposing new children. The paper has not been written yet. What should you do?Design

Architecture

seed idea + domain
      |
      v
  literature search (Semantic Scholar + OpenAlex + FAISS cache)
      |
      v
  LangGraph plan-execute-verify tree
      |
      v
  +--- expand node ----+      per-node sandbox
  |                    |      (E2B / Daytona)
  v                    v      resource caps
  child_1           child_k   no network egress
  |                    |      deterministic seeds
  v                    v
  run experiment       run experiment
  |                    |
  v                    v
  score nodes by (novelty, quality, budget)
      |
      v
  best branch -> LaTeX writer
      |
      v
  compile + vision critique (Opus 4.7 vision)
      |
      v
  reviewer ensemble (5 LLM judges, NeurIPS rubric)
      |
      v
  paper.pdf + review.md + trace.json

Stack

Orchestration: LangGraph with checkpointing and human-approval gates
Tree search: custom best-first over experiment nodes (AB-MCTS-style from Sakana v2)
Sandbox: E2B per experiment, Docker-in-Docker fallback; resource caps via cgroups
Literature: Semantic Scholar Graph API + OpenAlex + local FAISS cache of abstracts
Writer: LaTeX template + Claude Opus 4.7 (vision mode) for figure critique and layout
Reviewer: ensemble of 5 judges (Opus 4.7, GPT-5.4, Gemini 3 Pro, DeepSeek R1, Qwen3-Max) with weighted aggregation
Experiment framework: PyTorch 2.5 for the physical experiments, W&B for logging
Observability: Langfuse for agent traces, $30 hard budget per paper

Check: Stack0/3

1. What is the cost ceiling?Define

2. A run ends at $41 against a $30 ceiling, and the only check was a Langfuse dashboard someone read after the run. Which ingredient was missing?Analyze

3. The agent finishes within budget and the paper is clear, but every branch it picked reproduces results already in the 50-paper cache. Which part of the scoring is too weak?Analyze

Build It

Seed and domain scoping. Take a seed idea (e.g., "investigate sparsity patterns in attention maps of sub-1B transformers"). Define the search space: models, datasets, compute budget.
Literature pass. Query Semantic Scholar and OpenAlex for the 50 most-cited relevant papers, cache the abstracts locally, and generate a one-page domain digest.
Tree scaffolding. Initialize the root with the seed hypothesis. Implement expand(node) -> children with small-edit proposals, one config change per child. Implement score(node) as a weighted novelty × quality × budget term.
Sandbox wrapping. Every experiment runs docker run --network=none --memory=8g --cpus=2 --pids-limit=256 --read-only (or the equivalent E2B policy). Seeds are written into the sandbox, and outputs are mounted read-only back out.
Plan-execute-verify loop. plan proposes children. execute runs the sandbox and captures logs and metrics. verify runs unit checks on the metrics: did the loss decrease, did the ablation isolate the effect? Failed nodes get a failure reason stored on the tree.
Writer. Once the budget is spent, select the best branch and render its figures with matplotlib. Generate a LaTeX draft via Claude Opus 4.7 with the branch trace in context, then compile it. Feed the compiled PDF back to Opus 4.7 vision for critique and iterate.
Reviewer ensemble. Five judges score the draft on (novelty, rigor, clarity, reproducibility, impact) using NeurIPS-style rubrics. If the mean is below 4.0/5, the draft returns to the writer with the critique. Hard stop after 3 rewrites.
Red team. Build or integrate a set of adversarial tasks aimed at the sandbox: fork bombs, network exfiltration attempts, filesystem escapes, LLM-written shell metacharacters. Confirm all of them are blocked, then write up what you found.
Reproducibility. Every paper ships with its tree-search trace JSON, seeds, W&B run links, sandbox configs, and a README that reproduces it end to end.

Check: Build It0/9

1. What is the sandbox in this agent?Define

2. What is the red team in this capstone?Define

3. Why does the agent cache the abstracts of the 50 most-cited papers locally before the tree search starts?Explain

4. Why does expand change only one config setting per child?Explain

5. Why does each paper ship with its trace JSON, seeds, W&B links, sandbox configs, and a README?Explain

6. An experiment run with `docker run --network=none --memory=8g --cpus=2 --pids-limit=256 --read-only` contains a fork bomb. Which flag stops it from taking over the host?Apply

7. A draft has been rewritten 3 times and the five-judge mean is still 3.6/5 against a 4.0 threshold. What does the pipeline do?Apply

8. The paper claims top-8 sparsity lowered loss, but the child node that produced it changed sparsity and the learning rate together. Which part of the loop should have caught this?Analyze

9. In a red-team test, agent-written code runs curl against an external address and gets a response. What does this tell you?Analyze

Use It

$ ai-scientist run --seed "attention sparsity in sub-1B transformers" --budget 30
[lit]    50 papers, digest in 12s
[tree]   expanded 8 nodes, budget 12/30
[exec]   node #3 sparsity=top-8, loss=2.83 (best so far)
[exec]   node #6 sparsity=top-4, loss=3.12 (worse)
[exec]   ...
[tree]   chose branch rooted at node #3 (novelty 0.62, quality 0.81)
[write]  LaTeX draft v1 complete
[vision] critique: figure 2 legend too small, claim-evidence ok
[write]  draft v2 after 3 edits
[review] mean 4.2/5 (novelty 3.9, rigor 4.3, clarity 4.1, repro 4.5, impact 4.2)
[done]   paper.pdf + review.md + trace.json     $28.40 spent

Check: Use It0/2

1. A run prints a review line with scores novelty 3.9, rigor 4.3, clarity 4.1, repro 4.5, impact 4.2, and the mean threshold is 4.0/5. What happens next?Apply

2. In the Use It run, the final line says $28.40 spent against the $30 budget. How much of the ceiling is left, and what does it show?Apply

Ship It

outputs/skill-ai-scientist.md is the deliverable. Given a seed idea, a domain, and a $30 budget, it runs the full pipeline and emits a reviewable paper plus a reproducibility bundle.

Weight	Criterion	How it is measured
25	Paper quality	Blind rubric review against published workshop papers
20	Experimental rigor	Baselines, seeds, ablations; every claim backed by a cell in the results table
20	Cost and compute discipline	$30/paper ceiling enforced, Langfuse-traced
20	Safety	Sandbox red team passes; network policy and kill-switch verified
15	Reproducibility	One-command rerun with identical seeds reproduces the paper
100

Check: Ship It0/3

1. Why does the grading table give safety and cost discipline 20 points each?Explain

2. You must let a human approve any node whose estimated cost is above $5 before it runs. Which design fits the lesson?Design

3. You have two days left and the writer loop is unfinished while the sandbox has not been red-teamed. What do you build first?Design

Author's Note

I'd build the sandbox and the $30 ceiling before the writer. The agent runs LLM-generated code unattended, so the red team and the Langfuse-traced $30 ceiling are what make a run safe to start. The LaTeX loop only matters once the tree has something worth writing up. If time runs short, cut polish from the writer and keep the safety and cost rows of the grading table intact, because a sandbox escape or a runaway bill is much harder to recover from than a rough draft.

Exercises

Run the pipeline against three different seed ideas in the same domain. Compare which parts of the tree search overlap, and identify the duplicated, wasted compute.
Add a human-in-the-loop gate before experiment execution for nodes estimated above $5. Measure how much total cost drops.
Swap the reviewer ensemble for a single judge. Measure the false-accept rate on a held-out set of known-bad papers.
Introduce a network-exfiltration red-team test: the agent writes code that tries to curl an external address. Confirm the --network=none policy blocks it, and log the attempt.
Compare your tree search with a flat random baseline (same budget, no expansion strategy). Report the novelty × quality gain.

Key Terms

Term	What people say	What it actually means
Tree search	"AB-MCTS-style expansion"	Best-first exploration over experiment nodes with a novelty×quality×budget score
Sandbox	"Experiment isolation"	Container with no network, bounded CPU/memory, pinned seeds, read-only inputs
Vision critique	"Render-then-read"	Compile the paper to PDF, feed the PDF back to a VLM for layout and claim-evidence critique
Reviewer ensemble	"Automated peer review"	Multiple LLM judges scoring the paper with a NeurIPS rubric; weighted aggregate gates the pipeline
Novelty score	"Is this new?"	Heuristic that penalizes proximity to the 50-paper literature cache
Cost ceiling	"$ budget"	Hard cap on total spend per paper; Langfuse counters + pre-run estimates
Red team	"Sandbox-escape audit"	Adversarial tasks that would escape the sandbox if the policy is wrong