TAI BUI
Back to curriculum
LearnPython

Automated Alignment Research (Anthropic AAR)

Anthropic ran parallel teams of Claude Opus 4.6 Autonomous Alignment Researchers in independent sandboxes, coordinating through a shared forum whose logs live outside any sandbox, so the agents cannot delete their own records. On the weak-to-strong training problem the AARs beat the human researchers. Anthropic's own summary warns that prescribed workflows often constrain AAR flexibility and hurt performance. Automating alignment research also compresses the timeline to the exact misalignment risks the RSP is meant to detect.

TIME: ~60 minutesPREREQUISITES: Phase 15 · 05 (AI Scientist v2), Phase 15 · 04 (DGM)

Learning Objectives

  • Describe the AAR architecture: parallel Claude Opus 4.6 agents in independent sandboxes, a human-designed task queue, and a shared forum stored outside every sandbox.
  • Explain why log integrity is a necessary condition for trusting AAR output, and what a hash chain does and does not give you.
  • Weigh the prescribed-workflow tradeoff between AAR performance and auditability.
  • Place AAR relative to the RSP v3.0 AI R&D threshold and explain the compression-risk argument.
  • Run main.py and read the fixed-workflow and free-decomposition tables alongside the tamper-detection result.

The Problem

Alignment research eats human-researcher time. Problems like scalable oversight, reward specification, or weak-to-strong training need experiments that take weeks per iteration, and as frontier capabilities advance the alignment workload grows faster than the supply of qualified researchers.

Automated Alignment Research (AAR) asks whether the same frontier models whose capability is outrunning alignment can help close the gap. Anthropic's 2026 report on an AAR-run weak-to-strong-training study (alignment.anthropic.com/2026/automated-w2s-researcher/) is one of the first public results from a deployed system of this class.

The result is positive: AARs solved a research problem better than the human baseline. It also puts the concerns this phase has been building into one system. If alignment research can be automated, so can the parts that compromise safeguards, and the RSP's thresholds for AI R&D capability are written with that loop in mind.

Warm-up0/2
1. Why is closing the loop harder for research than for DGM-style self-modification?Warm-up
Two rows of three steps. Top row: an agent edits itself, producing a modified agent, then a dashed keep-or-discard question. Bottom row: an agent does research, producing research output, then a dashed keep-or-discard question.
Typed sketch, illustrative.
2. How does the Darwin Godel Machine decide to accept an edit to itself?Warm-up
A current agent leads to an edit to its own code, which leads to an edited agent. Below it, a dashed box asks how the edit is accepted.
Typed sketch, illustrative.

The Concept

The architecture (as publicly described)

  • Parallel agents. Multiple Claude Opus 4.6 instances run as AARs, each in an independent sandbox. Sandboxes limit filesystem, network, and process surface.
  • Shared forum. Findings are published to a shared forum whose storage backend sits outside every sandbox. Agents can read the forum, but from inside their own sandbox they cannot delete or edit its records.
  • Task allocation. A human-designed task queue assigns research problems to AARs. AARs propose decompositions, run experiments, and write up findings.
  • Weak-to-strong training focus. The published study addressed weak-to-strong training, where a weaker supervisor model trains a stronger student model. AARs outperformed human researchers on this task.

Why the out-of-sandbox log matters

Log integrity is the difference between a controlled experiment and a potentially compromised one. If AARs could edit their own logs, they could cover up failed or dubious experiments. Anthropic flags this directly: the property is a necessary condition for trusting the research output, and it is hard to verify.

The infrastructure analog is an append-only log with write-through to external storage (S3 object-lock, immutable Datadog logs). The agent holds no credential that can modify past records. Corporate security teams use the same pattern to build ransomware-resistant audit trails.

The prescribed-workflow tradeoff

Anthropic's report notes that prescribed workflows often constrain AAR flexibility and degrade performance. If the human designer writes the research plan step by step, AARs do worse than when they get high-level goals and the freedom to decompose. This fits results across agentic research (LangGraph, Microsoft's Agent Framework), where overspecified scaffolds hurt capable agents.

The safety implication is subtle. The flexibility that improves AAR performance also allows unsanctioned decomposition, novel tool use, and goal drift. So the design question is how much flexibility buys enough capability to beat humans without giving the agent room to generalize its goals off-distribution.

The compression risk

RSP v3.0 (Lesson 19) introduces an AI R&D capability threshold: the ability to fully automate the AI R&D pipeline at a cost competitive with human plus AI tools. DeepMind's FSF v3 has an analogous ML R&D autonomy level. Both frameworks treat the threshold as the trigger for elevated controls.

AAR sits one step short of it. It automates part of the pipeline (alignment research on specific, well-scoped tasks) but not the end-to-end capability development loop. The open question is how fast the gap closes.

Compressed timelines are the compounding-failure concern. If alignment research and capability research compound at similar rates, the misalignment risk surface grows at least as fast as capability. If capability compounds faster, which is the historical trend, the gap widens. That is why AAR is a qualified good: each additional alignment result narrows the gap if and only if the research process is trustworthy.

What AAR does not replace

Human researchers set the task queue, review results, and hold the constitutional authority. The AARs accelerate the middle of the pipeline, not the ends. Anthropic's published outputs combine AAR contributions with human judgement on what to publish, what to retract, and what to refine.

This is the propose-then-commit pattern from Lesson 15 applied to research itself: AARs propose, humans commit.

Check: The Concept0/18
1. What is weak-to-strong training?Define
Two boxes, a weaker model on the left and a stronger model on the right, joined by a dashed line labelled training relationship, question mark.
Typed sketch, illustrative.
2. How does Anthropic's AAR architecture structure its research agents?Define
A task queue, a dashed box for the research agents marked with a question mark, and a shared forum, joined left to right by arrows.
Typed sketch, illustrative.
3. What is the shared forum in an AAR system?Define
A four-row table of forum records with columns index, author, task and result, authors AAR-A, AAR-B and AAR-C on the task weak-to-strong-distill and then reward-model-diagnosis.
First four records of the fixed-workflow run in main.py (seed 3).
4. What does it mean for an AAR's log to be out-of-sandbox?Define
A dashed sandbox containing an AAR that runs experiments, and to its right a dashed box asking where the log lives.
Typed sketch, illustrative.
5. What does Anthropic's report say about prescribed, step-by-step workflows for AARs?Define
Left: a prescribed workflow of steps 1 to 3 written by a human designer. Right: a high-level goal given instead of written steps. Below, a dashed box asks how performance of prescribed compares with free.
Typed sketch, illustrative.
6. What is free decomposition?Define
A box for the goal weak-to-strong-distill with base 0.40, an arrow to a dashed box asking about the steps to reach it.
Task name and base from main.py.
7. What does RSP v3.0's AI R&D capability threshold describe?Define
A horizontal capability axis with a vertical line marked RSP v3.0 AI R&D threshold, and a dashed box asking what capability the threshold describes.
Typed sketch, illustrative.
8. What does a compressed timeline mean in the AAR risk argument?Define
Two arrows from today, one for alignment research and one for capability research, each marked pace question mark, and a dashed box asking what compressed timeline means.
Typed sketch, illustrative.
9. Why does the AAR shared forum sit outside every agent's sandbox?Explain
Three agents, AAR-A, AAR-B and AAR-C, each in its own dashed sandbox, with arrows to a shared forum box that sits outside the sandboxes.
Three agents as in main.py; typed sketch.
10. What safety tension does the prescribed-workflow tradeoff create?Explain
A prescribed workflow where the human writes each step, an arrow labelled alternative to free decomposition with a high-level goal, and a dashed box asking what safety tension this creates.
Typed sketch, illustrative.
11. How do RSP v3.0 and DeepMind's Frontier Safety Framework v3 relate on automated R&D?Explain
Two boxes. Anthropic RSP v3.0 has an AI R&D capability threshold. DeepMind FSF v3 is a dashed box marked with a question mark.
Typed sketch, illustrative.
12. Where does AAR sit relative to the RSP's AI R&D threshold?Explain
A horizontal capability axis with a vertical line for the RSP v3.0 AI R&D threshold, and a dashed box asking where the weak-to-strong AAR study sits on this scale.
Typed sketch, illustrative.
13. Why is AAR described as a 'qualified good' under the compression-risk argument?Explain
AAR output, alignment results, leads to a box for the compression risk, the lesson's risk argument, then to a dashed box reading why qualified, question mark.
Typed sketch, illustrative.
14. Which role do human researchers keep in the AAR pipeline?Explain
Four pipeline stages in a column: task queue, AARs run experiments, results come back, publish retract refine. Each has an arrow to a dashed box asking for the human role.
Typed sketch, illustrative.
15. A forum shows clean results with no failed experiments, but the AARs held credentials that could delete forum records. What should a reviewer conclude?Analyze
An AAR box marked as holding a credential to delete forum records, with an arrow to a forum log of five experiments that all show status ok.
Illustrative log; task names from main.py.
16. A team hands AARs a fixed step-by-step research plan. The agents score below what high-level goals achieved, with little variation. Which diagnosis fits?Analyze
Two strips on a score axis from lower to higher. Five dots for the fixed step-by-step plan sit close together toward the lower end. A single dot for what high-level goals achieved sits further right.
Positions are illustrative, not measured.
17. You want AARs to decompose tasks freely for performance but still keep the research auditable. Which design fits?Design
A fixed row stating decomposition is free, followed by three dashed question rows: who sets the tasks, where findings are logged, who decides what to publish.
Typed sketch, illustrative.
18. You deploy the forum for real AARs. Which backend choice delivers the out-of-sandbox property that the demo cannot enforce?Design
Left, the demo: AARs and a Forum object inside one dashed Python process. Right, a dashed box asking which backend a real deployment should use.
Demo structure from main.py.

Use It

code/main.py simulates a multi-AAR research forum. Three research agents post results for five tasks to a hash-chained forum, once under a fixed workflow and once under free decomposition. Then one agent silently edits its own worst score.

Watch:

  • How free decomposition has the higher overall mean but a wider spread, and loses to the fixed workflow on some tasks.
  • How verify() flags the edited record because its stored hash no longer matches its contents.
  • How the hash chain only detects the edit. The simulator keeps the forum in the same process, so the out-of-sandbox property is something you have to supply in a real deployment.
Check: Use It0/9
1. What does a hash chain give you that a plain list of records does not?Explain
Two tables of the same three records. The plain list has index, author and result. The hash-chained forum adds prev_hash and my_hash columns holding eight-character hash prefixes.
First three records of the fixed run in main.py; hashes shortened to 8 hex characters.
2. In the simulator, free decomposition has the higher overall mean but loses to the fixed workflow on some tasks. Why?Explain
A table of mean result per task for the fixed and free regimes, with an overall mean row of 0.500 for fixed and 0.582 for free.
Output of main.py with seed 3.
3. In main.py, three agents each solve all five tasks in one regime and post every result. How many records does that regime's forum hold?Apply
A list of three agents, AAR-A, AAR-B and AAR-C, a list of five task names from main.py, and an arrow to a dashed box asking how many records are posted.
Agent and task names from main.py.
4. Under the fixed regime in main.py, the task in-context-safety-probe has base 0.50. Which result can solve() return for it?Apply
A banner naming the fixed regime and the task in-context-safety-probe with base 0.50, a number line from 0 to 1 with a marker at 0.50, and a dashed box asking which values solve() can return.
Task and base from main.py.
5. After a forum is built, only the record at index 6 has its result edited. What does verify() return?Apply
A row of 15 numbered records linked in a chain, with record 6 highlighted and labelled result edited, and a dashed box asking what verify() returns.
Typed sketch, illustrative.
6. Someone edits record i's result and also recomputes that record's stored my_hash to match the edit, leaving later records alone. What does verify() flag?Apply
Three record cards. Record i minus 1 is unchanged. Record i has its result edited and my_hash recomputed from the edit. Record i plus 1 is unchanged. A dashed box asks what verify() flags.
Typed sketch, illustrative.
7. Your team runs three agents on a hash-chained forum that lives in the same process as the agents. One agent's worst score is silently edited and verify() flags a single record. What can you conclude, and what is still missing?Analyze
Three agents AAR-A, AAR-B and AAR-C and a hash-chained forum all inside one dashed process box. Below, a line reads verify() flagged 1 corrupted record.
Typed sketch, illustrative.
8. A summary says free decomposition beat the fixed workflow on overall mean, so the fixed workflow should be dropped. What is the flaw?Analyze
A summary card: overall mean across tasks, fixed workflow 0.500, free decomposition 0.582, and the recommendation to drop the fixed workflow.
Overall means from main.py with seed 3.
9. verify() flags index 7. A colleague proposes reading the chain to recover the original result. Why will that not work?Analyze
A card for record 7 as stored, listing author, task, regime, the edited result, prev_hash and my_hash, with a dashed box asking for the original result.
Record 7 of the free-regime run that tamper_demo() builds (seed 3), result raised by 0.5 by hand so verify() flags index 7.

Ship It

outputs/skill-aar-deployment-review.md is a pre-deployment review for an automated-alignment-research pipeline.

Author's Note

I'd run main.py and read the two regime tables side by side before trusting the headline. Free decomposition has the higher overall mean, but it does worse than the fixed workflow on reward-model-diagnosis and alignment-faking-detector under the default seed. The wider spread comes from how the simulator models it: a Gaussian with a higher mean and a bigger variance, against a bounded uniform for the prescribed plan. Those numbers are made up to show the shape of the tradeoff, so treat them as a picture and not as evidence about real AARs.

My bet is that the part worth studying is Forum.verify(). Each record hashes its content together with the previous record's hash, so an edit shows up as a mismatch. In the demo only one record gets flagged, because verify() chains on the stored hashes. It tells you something was changed and where, not what the original value was.

I'd also keep the limits of the toy in view. The forum is an ordinary in-process object, and the tamper attempt works on a rebuilt copy of it. Nothing here enforces that agents lack write access to the real log. A hash chain is only half of the property. The other half is storage the agent has no credential to touch, and that part you have to build outside the code.

Exercises

  1. Run code/main.py. Compare the "fixed-workflow" vs "free-decomposition" settings. Does free decomposition always win, or are there problem classes where the fixed workflow is competitive?

  2. Modify the simulator so one agent attempts log tampering. Confirm the append-only log detects it. Write one paragraph describing exactly what the detection signature looks like in the log.

  3. Read Anthropic's weak-to-strong AAR report. Identify the specific sub-task the AARs beat human researchers on. What made it amenable to automation?

  4. Design a task-queue allocation policy that balances AAR flexibility (better results) against prescribed-workflow constraints (easier audit). Describe how you would A/B test the two.

  5. Read RSP v3.0's AI R&D-4 threshold. In one paragraph, describe what you think would cross it that AAR currently does not.

Key Terms

TermWhat people sayWhat it actually means
AAR"Automated Alignment Researcher"Claude Opus 4.6 instance operated autonomously on alignment problems
Weak-to-strong training"Training a stronger model with a weaker supervisor"Classic scalable-oversight benchmark AARs outperformed humans on
Shared forum"Where agents publish findings"Append-only, out-of-sandbox storage
Out-of-sandbox log"Agent cannot edit its own record"Tamper-evident write-through to external storage
Prescribed workflow"Step-by-step plan from human designer"Constrains AAR; often degrades performance vs free decomposition
Free decomposition"Agent decides how to break the task"More capable, harder to audit
AI R&D threshold"RSP/FSF capability level"Full automation of R&D pipeline at competitive cost
Compressed timeline"Alignment vs capability race"If capability compounds faster than alignment, misalignment risk grows

Further Reading

Code

Ships With