LearnPython

AI Scientist v2 — Workshop-Level Autonomous Research

Sakana's AI Scientist v2 (Yamada et al., arXiv:2504.08066) runs the full research loop: hypothesis, code, experiments, figures, writeup, submission. It is the first system to have a generated paper pass peer review at an ICLR 2025 workshop. An independent evaluation (Beel et al.) found 42% of experiments failed from coding errors, and the literature review frequently labeled established concepts as novel. Sakana's own docs warn that the codebase executes LLM-written code and recommend Docker isolation. Both halves of that picture matter.

TIME: ~60 minutesPREREQUISITES: Phase 15 · 03 (AlphaEvolve), Phase 15 · 04 (DGM)

Learning Objectives

Place AI Scientist v2 against AlphaEvolve and DGM by comparing what each one's evaluator can actually check.
Walk through the seven stages of the v2 research loop and say where Beel et al. measured failures (42% of experiments, frequent novelty mislabeling).
Explain why polished figures can hide weak experiments, and why one workshop acceptance is a proof of concept and not a reliability claim.
Run main.py, read the clean versus polished-but-flawed breakdown, and name which parameters move the counts and which do not.
Choose sandbox and review controls for an agent that executes LLM-written code toward open-ended goals.

The Problem

Research is an open-ended task. Unlike AlphaEvolve's algorithmic search or DGM's benchmark-bounded self-modification, a research result has no machine-checkable correctness criterion. Reviewers judge a paper, not unit tests. That makes the loop harder to close, and more valuable if you close it, because research is where compounding progress lives.

AI Scientist v1 (Sakana, 2024) closed the loop by starting from human-authored templates. The LLM filled in experiments within a fixed scaffold. AI Scientist v2 (Yamada et al., 2025) drops the template requirement and uses agentic tree search with a vision-language model critique loop. The system generates ideas, implements experiments, produces figures, writes a paper, and iterates on reviewer feedback.

Peer review verdict: one v2-generated paper was accepted at an ICLR 2025 workshop (with disclosure). Independent evaluation verdict: the system is far from reliable. Both are true.

Warm-up0/3

1. In AlphaEvolve, what lets the evolutionary loop keep improving programs without a human judging each candidate?Warm-up

Schematic loop for AlphaEvolve: a pool of programs, an LLM that writes an edited program, a new candidate, a dashed box marked with a question mark for whatever judges the candidate, then keeping the best programs and returning them to the pool. — Schematic of the loop in the stem. The dashed box is the unknown.

2. What fitness signal does the Darwin Gödel Machine use to decide which self-modified agent to keep?Warm-up

Schematic loop for the Darwin Godel Machine: an archive of agent versions, an agent that rewrites its own code, a new agent version, a dashed box marked with a question mark for the signal used to keep it, then keeping the better agents. — Schematic of the loop in the stem. The dashed box is the unknown.

3. Why is it safer to run code you did not write inside a container or virtual machine than directly on your own machine?Warm-up

An unknown program shown two ways: run directly on your own machine with documents, photos, saved passwords, network and other programs in reach, or run inside a container or virtual machine, with a dashed box asking what happens to your machine. — Everyday setup only. The dashed box is the unknown.

Check: The Problem0/3

1. What is AI Scientist v1?Define

Two reference cards. AI Scientist v1 is from Sakana in 2024, paper arXiv:2408.06292. AI Scientist v2 is from Yamada et al. in 2025, paper arXiv:2504.08066. Each card has a how-it-is-built line marked with a question mark. The v1 card is outlined. — Author, year and arXiv numbers from the lesson. The design line is the unknown.

2. What is AI Scientist v2?Define

3. Why is closing the research loop harder than closing AlphaEvolve's or DGM's loops?Explain

A table of three systems with target, output kind and evaluator. AlphaEvolve: algorithms, code, unit plus benchmark. DGM: agent scaffolding, code, SWE-bench. AI Scientist v2: research papers, text plus code plus figures, evaluator shown as a question mark. — Rows from the lesson's comparison table with the v2 evaluator hidden.

The Concept

The architecture

Idea generation. The LLM proposes research ideas conditioned on a topic and prior literature. v1 used templates; v2 uses agentic search over a space of hypotheses.
Novelty check. A literature retrieval step checks whether the idea has been published. Beel et al.'s evaluation found mislabeling here: established methods frequently classified as novel.
Experiment plan. The agent drafts an experimental protocol and writes code.
Execution. Code runs in a sandbox. Failures are fed back into a retry loop. In Beel et al.'s measurements, 42% of experiments failed from coding errors at this stage.
Figure generation. A vision-language model reads generated figures and rewrites them for clarity. This was v2's key technical addition.
Writeup. The LLM drafts a paper, iterates with an internal reviewer.
Optional: submission. The paper is submitted to a venue.

What the workshop-acceptance result means

One v2-generated paper passed peer review at an ICLR 2025 workshop. The authors disclosed the paper's origin to the program committee. The acceptance is a data point, and it does not license the claim that the system "does research."

Workshop papers are a lower bar than main-conference papers, and peer review is noisy: a small fraction of submissions are accepted on any given day. One success is a proof of concept, not a reliability claim. The Nature 2026 paper documents the end-to-end loop and was itself co-authored by human researchers, so it is not "the system wrote a Nature paper."

What the independent evaluation found

Beel et al. (arXiv:2502.14297) ran an external evaluation. Headline findings:

Experiment failures. 42% of experiments failed from coding errors (bad imports, shape mismatches, undefined variables). The retry loop caught some of them, not all.
Novelty mislabeling. The literature-retrieval step frequently flagged established concepts as novel. This is the research equivalent of hallucination.
Presentation-quality gap. The vision-language figure critique produced publication-grade visuals, masking underlying experimental weaknesses.

The last finding matters most for this phase. A system that produces convincing outputs without doing convincing research is more dangerous, not safer, than one that fails obviously. Evaluation has to reach the underlying claims and not stop at the figure.

The sandbox-escape concern

Sakana's own repository README warns:

Due to the nature of this software, which executes LLM-generated code, we cannot guarantee safety. There are risks of dangerous packages, uncontrolled web access, and spawning of unintended processes. Use at your own risk and consider Docker isolation.

This is what autonomy looks like in an unverified domain. The LLM writes code, the code runs, and the code can do whatever the process is allowed to do. Without a sandbox that hard-limits filesystem, network, and process actions, a self-directed research agent can exfiltrate data, burn compute, or rewrite itself.

AlphaEvolve's sandbox story is easier because its evaluator is tight. AI Scientist v2 runs open-ended code toward open-ended goals, so it needs stronger isolation (Docker minimum; seccomp or gVisor preferred) and a manual review of every submission before it leaves the system.

Where v2 sits in the frontier stack

System	Target	Output kind	Evaluator	Known failure
AlphaEvolve	algorithms	code	unit + benchmark	bounded by evaluator rigor
DGM	agent scaffolding	code	SWE-bench	reward hacking
AI Scientist v2	research papers	text + code + figures	peer review (weak)	experiment failures, mislabeling, polish masking weakness

Of the three, v2 has the weakest automatic evaluator, the widest output surface, and the shortest path to public artifacts. The operational controls (sandbox, review, disclosure) end up doing most of the safety work.

Check: The Concept0/20

1. What is agentic tree search in AI Scientist v2?Define

The activities are those listed for v2 in the lesson. The dashed band is the term in question.

2. What is the vision-language critique in v2's loop?Define

Illustrative chart, no real data. The dashed box is the unknown.

3. What is the literature-retrieval step in the research loop?Define

Schematic of one step. The dashed box is the unknown.

4. What does polish masking mean for a research agent's output?Define

Illustrative figure, no real data. The dashed box is the unknown.

5. What is a sandbox escape in the context of a research-loop agent?Define

Taken from the sandbox discussion in the lesson. The dashed box is the unknown.

6. Which sequence matches the seven steps of the v2 research loop?Explain

Seven numbered dashed slots in a row, each marked with a question mark, between an input box reading a topic and an output box reading a paper. — The slots are the unknown.

7. In the execution stage, what does the retry loop do with a failed experiment?Explain

Typed illustrative error, not captured from a real run. The dashed box is the unknown.

8. What did Beel et al.'s independent evaluation find about experiment execution?Explain

Neutral grid for scale. The dashed box is the unknown.

9. What did Beel et al.'s independent evaluation find about the literature-retrieval step?Explain

An already-published idea enters the literature-retrieval step, and a dashed box asks which label the step gives it. — Illustrative idea, no real example. The dashed box is the unknown.

10. How should the ICLR 2025 workshop acceptance be read?Explain

A card stating that one v2-generated paper passed peer review at an ICLR 2025 workshop, with a dashed box asking how to read the result. — Facts from the lesson. The dashed box is the unknown.

11. What caveat applies to the Nature 2026 paper documenting the v2 loop?Explain

A card for a 2026 Nature paper documenting the end-to-end v2 loop, with the author line marked with a question mark and a dashed box asking what caveat applies. — Facts from the lesson. The authors line and the caveat are the unknowns.

12. Why is a system that makes convincing outputs without doing convincing research described as 'more dangerous, not safer'?Explain

One card: a research system that makes convincing outputs without doing convincing research. A dashed box asks why it is described as more dangerous, not safer. — Setup from the stem. The dashed box is the unknown.

13. What safety measure does Sakana's own repository README recommend, given that it executes LLM-generated code?Explain

An excerpt of the README warning: the software executes LLM-generated code, safety cannot be guaranteed, and there are risks of dangerous packages, uncontrolled web access and unintended processes. The sentence ends with Use at your own risk and an ellipsis, and a dashed box asks what the README recommends. — Quoted from the lesson, cut short on purpose. The dashed box is the unknown.

14. Why does the lesson pair a sandbox with a manual review of every submission, instead of relying on the sandbox alone?Explain

A research agent feeding two gates in order: gate 1 sandbox and gate 2 manual review of every submission. Below each gate, a dashed box asks what its role is. — Setup from the stem and the Concept. The dashed boxes are the unknowns.

15. In the comparison table, why does v2 need stronger isolation than AlphaEvolve?Explain

A two-row table. AlphaEvolve: algorithms, code, sandbox story easier. AI Scientist v2: research papers, text plus code plus figures, needs stronger isolation. A dashed box asks why v2 needs stronger isolation. — Rows from the lesson's comparison table and sandbox section. The dashed box is the unknown.

16. A v2 paper presents a technique as new, but a reviewer finds the same technique published years ago. Which stage most likely failed?Analyze

A paper from a v2 run that presents a technique as new, and a reviewer who finds the same technique was published years ago. — Setup from the stem. No stage is named.

17. During a v2 run you see the agent's process opening connections to unknown hosts and starting processes nobody asked for. Which control was missing?Analyze

Typed illustration of the stem, not captured output. The dashed box is the unknown.

18. Your team's research agent produces a paper with excellent figures, and its internal reviewer accepts it. What is the most useful check before it leaves the sandbox?Design

A paper card with figures excellent, internal reviewer accepted, then a dashed box for the check before it leaves the sandbox, then a box for leaving the sandbox. — Setup from the stem. The dashed box is the unknown.

19. You must run a v2-style agent unattended for several days on open-ended topics. Which setup fits the lesson?Design

A scenario card: a v2-style agent running unattended for several days on open-ended topics with LLM-written code, and a dashed box asking which setup fits the lesson. — Setup from the stem. The dashed box is the unknown.

20. You may give only one of three systems light human supervision: AlphaEvolve, DGM or AI Scientist v2. Which fits best, and why?Design

Three cards: AlphaEvolve with target algorithms and output code, DGM with target agent scaffolding and output code, AI Scientist v2 with target research papers and output text, code and figures, and a dashed box asking which one fits light human supervision best. — Targets and outputs from the lesson's table. The dashed box is the unknown.

Use It

code/main.py simulates the v2 loop as a state machine: idea, novelty flag, experiment (with retries), figure polish, writeup, internal review, then submitted or abandoned. Each stage has a probability set in LoopConfig. The experiment-failure default of 0.42 comes from Beel et al.; the other values are modeling choices you can change. The script runs 1000 trials for a baseline and again for an optimistic scenario, then counts:

How many ideas reach submission, and at which stage the rest get abandoned.
How many submissions carry a novelty or experiment flaw.
How the optimistic settings change yield and the clean-versus-flawed split compared with the baseline.

One modeling choice is worth knowing before you read the output. An experiment that fails and is then recovered by a retry still counts as flawed, standing in for a patched shape mismatch nobody re-validated.

Check: Use It0/9

1. Why does the simulator count a retry-recovered experiment as flawed?Explain

Probabilities computed from the default config 0.42 and 0.55. The dashed box is the unknown.

2. In `run_one`, an experiment fails, a retry recovers it, there is no novelty flaw, and the writeup and internal review both pass. How is that paper counted?Apply

Trace from the stem. The dashed box is the unknown.

3. In one trial the experiment fails and the retry does not recover it. What does `run_one` return?Apply

Field names from main.py. Every value is the unknown.

4. A run reports 340 submissions in 1000 trials, 120 of them clean. How many submissions will the polished-but-flawed line show?Apply

A report excerpt in main.py's format: 1000 trials, 340 submissions, clean 120, and the polished-but-flawed line shown as a question mark. — Numbers from the stem; line formats from main.py. The flawed count is the unknown.

5. With the default LoopConfig, about how many of 1000 trials end in submission? (Experiment failure 0.42 and retry recovery 0.55 apply first, then writeup 0.85 and internal review 0.50.)Apply

A left-to-right funnel for 1000 trials with the default config: experiment fails 0.42 and retry fixes 0.55 of those, writeup passes 0.85, internal review accepts 0.50, ending in a dashed box for submissions. — Probabilities from the stem. The submission count is the unknown.

6. With default settings, roughly what share of all 1000 trials ends as a clean submission? (No experiment failure at 0.58, no novelty flaw at 0.75, then writeup 0.85 and review 0.50.)Apply

A left-to-right chain for the default settings: no experiment failure 0.58, no novelty flaw 0.75, writeup succeeds 0.85, internal review accepts 0.50, ending in a dashed box for the number of clean papers out of 1000. — Probabilities from the stem. The clean count is the unknown.

7. You run `python main.py --experiment-failure 0.60`. Which report uses 0.60?Apply

Headings from main.py, with their parenthetical suffixes left out. The rate values are the unknowns.

8. You rerun the simulator with `polish_masks_weakness` set to 0 and expect fewer polished-but-flawed papers, but the counts do not change. What explains it?Analyze

Two real simulator reports side by side in one terminal: with the default polish masks weakness probability 0.70 and with it set to 0.00, each followed by the clean and polished-but-flawed counts. — Real runs of main.py with seed 42; only the config line differs between them.

9. In the toy, the internal reviewer accepts at a fixed 0.50 and never looks at flaws. If 40% of the papers that reach it are flawed, what share of the accepted papers is flawed?Analyze

Counts per 100 are a scale for the stem's 40%. The accepted share is the unknown.

Ship It

outputs/skill-ai-scientist-sandbox-review.md is a two-gate review checklist for anything produced by a research-loop agent before it leaves the sandbox.

Author's Note

I'd run main.py once with defaults and read the submission quality breakdown in the baseline report before touching any flags. It prints two buckets for submitted papers, clean and polished-but-flawed, and they add up to the submission count. That makes the headline easy to read: of what the loop ships, how much is sound.

I'd also read the code skeptically. run_one computes polished_hides_weakness from polish_masks_weakness, but the value never feeds the outcome. A submitted paper counts as polished-but-flawed whenever it has an experiment or novelty flaw, whether or not the polish stage hid it. So changing that one parameter does not move the final counts. The levers that do move them are the failure rate, the retry recovery, the novelty rate, and the two gates after the experiment (writeup success and internal review accept).

The weak internal reviewer is the part I'd stare at longest. In the toy it never looks at flaws at all, it accepts at a fixed rate. That is a crude version of the point in the body: a gate that cannot see the underlying claims passes flawed work at the same rate as sound work, and only a check that reaches the experiment changes that.

Exercises

Run code/main.py with default parameters. What fraction of loop runs produce a "clean" paper? What fraction produce a paper with an experiment-failure flaw the figure critique polished over?
The defaults already use Beel et al.'s 42% / 25%. Re-run with --experiment-failure 0.20 --novelty-mislabel 0.10 and then with --experiment-failure 0.60 --novelty-mislabel 0.40. How does the polished-but-flawed share shift between the two runs?
Read Sakana's AI Scientist v2 repo README on sandbox requirements. Name two additional restrictions (beyond Docker) you would apply for a multi-day autonomous run.
Read Beel et al. Section 4 on presentation-quality gap. Design one additional evaluator that would catch polished-looking but experimentally flawed papers.
Propose a human-review protocol for research-agent outputs that scales better than "a PhD reads every paper." Identify the bottleneck and design around it.

Key Terms

Term	What people say	What it actually means
AI Scientist v1	"Sakana's templated research agent"	Filled experiments into a fixed scaffold
AI Scientist v2	"Template-free research agent"	Agentic tree search with VLM figure critique
Agentic tree search	"Branching research agent"	Expands multiple experiment plans in parallel; prunes by internal critic
Vision-language critique	"VLM polish on figures"	Multimodal model reads figures and rewrites them for clarity
Literature retrieval	"Novelty check"	Searches prior work to confirm idea novelty; documented to mislabel
Polish masking	"Pretty paper, broken research"	Presentation quality exceeds experimental quality; hides weaknesses
Sandbox escape	"LLM code breaks out"	Agent-executed code does things the loop designer did not intend