CapstonePython / TypeScript (agents), Shell (worktree scripts)

Capstone 10 — Multi-Agent Software Engineering Team

A multi-agent software team is an architect, parallel coders, a reviewer, and a tester passing typed messages over a shared task board. SWE-AF, MetaGPT, AutoGen 0.4, Cognition's Devin, and Factory's Droids all landed on that shape in 2026. Parallel worktrees turn wall-clock time into throughput, and the handoffs between roles become the failure surface. Build the team, evaluate it on SWE-bench Pro, and report which handoffs break and how often.

TIME: 40 hoursPREREQUISITES: Phase 11 (LLM engineering), Phase 13 (tools), Phase 14 (agents), Phase 15 (autonomous), Phase 16 (multi-agent), Phase 17 (infrastructure)

Learning Objectives

Design a four-role software team (architect, parallel coders, reviewer, tester) where each role has its own model, permissions, and typed messages on a shared task board.
Isolate parallel coders in git worktrees plus Daytona sandboxes, and merge their branches with a three-way merge coordinator that calls an LLM only when files overlap.
Gate reviews so the reviewer cannot approve its own diffs, and measure its false-approval rate with an injected-bug probe.
Measure token amplification, pass@1, and cost per solved issue against a single-agent baseline on 50 SWE-bench Pro issues, then write a handoff-failure post-mortem.

Problem

A multi-agent software team is a set of role-specialized agents (architect, coders, reviewer, tester) that split one engineering task between them and coordinate through typed messages. Single-agent coding harnesses hit a ceiling on large tasks, and not because any individual agent is weak. A 200k-token context cannot hold an architecture plan, four parallel codebase slices, reviewer commentary, and test output at once. The factory shape fixes that by giving each concern its own context: the architect owns the plan, coders implement in parallel worktrees, the reviewer gates, and the tester verifies. SWE-AF's "factory" architecture, MetaGPT's roles, and AutoGen's typed actor graph are three framings of that one design.

The failure surface is the handoff. The architect plans something the coders cannot implement. Coders produce conflicting diffs. The reviewer approves a hallucinated fix. The tester races a coder that is still writing. You will build one of these teams, run it on 50 SWE-bench Pro issues, track every handoff, and publish the post-mortem.

Warm-up0/4

1. An agent works on a large coding task and its context window keeps filling up with plans, file contents, and tool output. What is the usual consequence?Warm-up

2. Two engineers edit the same lines of one file on separate git branches, then merge both. What does git do?Warm-up

3. An agent runs code it generated itself. Why run that code in a sandbox rather than on the host?Warm-up

4. Several agents pass work to each other. Why use structured, typed messages instead of free-form text?Warm-up

Check: Problem0/2

1. What is a handoff in a multi-agent software team?Define

2. Why split the task across several role-specialized agents instead of using one agent with a big context?Explain

Concept

Each role is a typed agent with its own model and its own permissions.

Architect (Claude Opus 4.7) reads the issue, writes a plan, and breaks it into subtasks with explicit interfaces.
Coders (Claude Sonnet 4.7, N parallel instances, each in a git worktree plus a Daytona sandbox) implement the subtasks independently.
Reviewer (GPT-5.4) reads the merged diff and either approves it or requests specific changes.
Tester (Gemini 2.5 Pro) runs the test suite in isolation and reports pass/fail with artifacts.

Roles communicate through a shared task board, either file-backed or Redis. Each role consumes only the tasks it is permitted to handle, and handoffs are A2A-protocol-typed messages. Three coordination concerns need explicit answers. Merge conflicts go to a coordinator role or an automatic three-way merge. Shared state needs a rule, and the rule here is that the plan freezes once coders start, with replans treated as separate events. Reviewer gatekeeping means the reviewer cannot approve its own changes or changes it proposed.

Token amplification is the hidden cost. Every role boundary adds summary prompts and handoff context, so a 40-turn single-agent run becomes 160 total turns across four roles. That is why the rubric weighs token efficiency against a single-agent baseline. The question is not whether multi-agent works. The question is whether it wins per dollar.

Check: Concept0/12

1. What is a parallel worktree in this team design?Define

2. What is the task board that the roles coordinate through?Define

3. What does token amplification measure?Define

4. What is the merge coordinator?Define

5. What constraint keeps the reviewer from rubber-stamping its own work?Define

6. Why does each coder work in its own git worktree plus a Daytona sandbox?Explain

7. The plan freezes once coders start. What happens if the architect later wants to change it?Explain

8. Why does a 40-turn single-agent run become about 160 turns across four roles?Explain

9. Your four-role team passes 4 points more issues than the single-agent baseline but uses about three times the tokens per solved issue. What is the right conclusion?Analyze

10. Coder B's diff calls a helper that coder A's subtask never exposes. Each coder's tests pass locally, but the merged branch fails. Which handoff broke?Analyze

11. Part of a diff was proposed by the reviewer's own model in an earlier round, and the reviewer now approves the whole diff. Which rule does this violate?Analyze

12. Midway through a run, coder C finds a wrong interface and edits the shared plan directly so the other coders see it. What went wrong?Analyze

Architecture

GitHub issue URL
      |
      v
Architect (Opus 4.7)
   reads issue, produces plan with subtasks + interfaces
      |
      v
Task board (file / Redis)
      |
   +-- subtask 1 ---+-- subtask 2 ---+-- subtask 3 ---+-- subtask 4 ---+
   v                v                v                v                v
Coder A          Coder B          Coder C          Coder D          (4 parallel)
 (Sonnet)         (Sonnet)         (Sonnet)         (Sonnet)
 worktree A       worktree B       worktree C       worktree D
 Daytona          Daytona          Daytona          Daytona
      |                |                |                |
      +--------+-------+-------+--------+
               v
           merge coordinator  (three-way merge + conflict resolution)
               |
               v
           Reviewer (GPT-5.4)
               |
               v
           Tester  (Gemini 2.5 Pro)  -> passes? -> open PR
                                     -> fails?  -> route back to coder

Stack

Orchestration: LangGraph with shared state + per-agent sub-graphs
Messaging: A2A protocol (Google 2025) for typed inter-agent messages
Models: Opus 4.7 (architect), Sonnet 4.7 (coders), GPT-5.4 (reviewer), Gemini 2.5 Pro (tester)
Worktree isolation: git worktree add per coder + Daytona sandbox
Merge coordinator: custom three-way merge + LLM-mediated conflict resolution
Eval: SWE-bench Pro (50 issues), SWE-AF scenarios, HumanEval++ for unit tests
Observability: Langfuse with role-tagged spans, per-agent token accounting
Deployment: K8s with each role as a separate Deployment + HPA on backlog

Check: Stack0/2

1. What is the A2A protocol used for in this stack?Define

2. Coder work backs up during a burst of issues while the reviewer and tester sit mostly idle. How does the stack absorb that?Design

Build It

Task board. Build a file-backed JSONL log of typed messages: plan_request, subtask, diff_ready, review_needed, test_needed, approved, rejected, replan_needed. Agents subscribe to tags.
Architect. Read the GitHub issue and run Opus 4.7 with a plan template that requires explicit subtask interfaces (files touched, public functions, test impact). It emits one plan_request containing a DAG of subtasks.
Coders. Run N parallel workers, each claiming one subtask from the board. Each spawns a fresh git worktree add branch and a Daytona sandbox, implements its subtask, and emits diff_ready with the patch and test deltas.
Merge coordinator. Once all coders finish, three-way merge the N branches into a staging branch. Call the LLM for conflict resolution only when file-level overlap exists.
Reviewer. GPT-5.4 reads the merged diff and cannot approve diffs it authored. It emits approved (a no-op) or review_feedback with specific change requests routed back to the relevant coder.
Tester. Gemini 2.5 Pro runs the test suite in a clean sandbox and captures artifacts. It emits test_passed or test_failed with stacktraces, and failed tests loop back to the coder who owns the failing subtask.
Handoff accounting. Every message crossing a role boundary gets a span in Langfuse with payload size and model used. Compute per-subtask token amplification as (coder_tokens + reviewer_tokens + tester_tokens + architect_share) / coder_tokens. The reference main.py reports the aggregate form instead: total team tokens divided by total single-agent baseline tokens across matched trials.
Eval. Run the 50 SWE-bench Pro issues. Compare pass@1 and $-per-solved-issue against a single-agent baseline: one Sonnet 4.7 in a single worktree.
Post-mortem. For each failed issue, name the handoff that broke (plan too vague, merge conflict, reviewer false-approve, tester flake) and produce a handoff-failure histogram.

Check: Build It0/19

1. For each failed issue in the post-mortem, what do you produce?Define

2. Why does the merge coordinator call the LLM only when file-level overlap exists?Explain

3. Why does the lesson say to get the single-agent baseline running first?Explain

4. Why does Build It tag every cross-role message with a trace span carrying payload size and model?Explain

5. The tester reports failing tests, and the stack trace points into the migration subtask. Where does the failure go?Apply

6. A team run uses 10,000 coder tokens, 3,000 reviewer tokens, 2,000 tester tokens, and a 5,000-token architect share. What per-subtask token amplification does the Build It formula give?Apply

7. In main.py, a diff has has_bug set and the reviewer stub's random draw is 0.90. What do approved and tested_passed hold in the returned result?Apply

8. In the main.py 10-trial comparison, the team averaged 34,819 tokens per run and the baseline 21,705. What amplification does it print?Apply

9. In that comparison the team solved 7 of 10 at 34,819 tokens per run, and the baseline solved 5 of 10 at 21,705. Which has the lower tokens per solved issue?Apply

10. launchWorktree receives the command python3 with argv ['-c', 'print(1); rm -rf x']. What happens?Apply

11. Which call does refuseReason refuse because of the command denylist?Apply

12. Coordinator.run starts with a message to the planner and maxTurns of 1. The planner replies with a message that is not review.approved. What does run return?Apply

13. A SharedWorkspace gets write('a.py', 'x', 'coder') and then write('a.py', 'y', 'reviewer'). What does read('a.py') return?Apply

14. The reviewer approved a diff that contains an early return None, and the tester then fails it. What does the post-mortem record?Analyze

15. You must show whether the four-role team beats one agent at all. Which evaluation design fits?Design

16. You suspect parallelism adds coordination cost without a real speedup. Which experiment isolates it?Design

17. You want to drop the merge coordinator entirely. Which design lets you do that, and what does it cost?Design

18. Your reviewer has a high false-approval rate and prompt tuning has stalled. Which next step follows the lesson?Design

19. You want to add a documenter role that writes a changelog entry after review. How should you decide whether to keep it?Design

Use It

$ team run --issue https://github.com/acme/widget/issues/842
[architect] plan: 4 subtasks (parser, cache, api, migration)
[board]     dispatched to 4 coders in parallel worktrees
[coder-A]   subtask parser  -> 42 lines, tests pass locally
[coder-B]   subtask cache   -> 88 lines, tests pass locally
[coder-C]   subtask api     -> 31 lines, tests pass locally
[coder-D]   subtask migration -> 19 lines, tests pass locally
[merge]     3-way merge: 0 conflicts
[reviewer]  comments on cache (thread pool sizing); routed to coder-B
[coder-B]   revision: 92 lines; submits
[reviewer]  approved
[tester]    all 412 tests pass
[pr]        opened #3382   4 coders, 1 revision, $4.90, 18m

Check: Use It0/1

1. In the Use It transcript, the merge reports 0 conflicts and the reviewer comments on one subtask. What does the final PR line count as revisions, and who made it?Apply

Ship It

outputs/skill-multi-agent-team.md is the deliverable. Given an issue URL and a parallelism level, the team produces a merge-ready PR with per-role token accounting.

Weight	Criterion	How it is measured
25	SWE-bench Pro pass@1	Matched 50-issue subset, pass@1
20	Parallel speedup	Wall-clock vs single-agent baseline
20	Review quality	False-approval rate on injected-bug probe
20	Token efficiency	Total tokens per solved issue vs single-agent
15	Coordination engineering	Merge-conflict resolution, handoff-failure histogram
100

Check: Ship It0/5

1. What is a false approval?Define

2. Why does the injected-bug probe give a clean measure of the reviewer's quality?Explain

3. Why does the rubric score token efficiency per solved issue rather than total tokens?Explain

4. With the same pass@1 as the baseline, the team finishes in half the wall-clock time but spends four times the tokens per solved issue. How does that score on the rubric?Analyze

5. Your reviewer approves 20% of diffs that contain an injected bug. What do you do first?Analyze

Author's Note

My bet is that you should get the single-agent baseline running before you write the architect. Step 8 defines it as one Sonnet 4.7 in a single worktree, and two criteria in the rubric, parallel speedup and token efficiency (20 points each), are both measured against it. If the baseline only shows up at the end, you will have no speedup or tokens-per-solved-issue comparison until the last day, and by then you have no room to react if multi-agent turns out to lose per dollar.

I'd also build the injected-bug probe from Exercise 1 before tuning anything else on the team. The Problem section names the reviewer approving a hallucinated fix as one of the handoff failures, and the review-quality criterion scores the false-approval rate directly. A reviewer that waves bugs through leaves the tester as the last gate before a bad PR, so I'd rather know that rate early.

Exercises

Inject an obvious bug into a diff mid-run (an extra return None before the main body). Measure the reviewer's false-approve rate, then tune the reviewer prompt until false-approval is under 5%.
Reduce to a single coder: architect, coder, reviewer, and tester, with the coder running the plan's subtasks sequentially. Compare wall-clock and pass rate.
Replace the merge coordinator with a single-writer constraint, so subtasks touch disjoint file sets. Measure the planning burden this puts on the architect.
Swap the reviewer from GPT-5.4 to Claude Opus 4.7. Measure the change in false-approval rate and token cost.
Add a fifth role, a documenter (Haiku 4.5), that produces a changelog entry after review. Measure whether the documentation quality justifies the extra token spend.

Key Terms

Term	What people say	What it actually means
Parallel worktree	"Isolated branch"	`git worktree add` producing a fresh working tree per coder
Task board	"Shared message bus"	File or Redis store of typed messages agents subscribe to
Handoff	"Role boundary"	Any message crossing from one role's context to another's
Token amplification	"Multi-agent overhead"	Total tokens across roles / single-agent tokens for the same task
A2A protocol	"Agent-to-agent"	Google's 2025 spec for typed inter-agent messages
Merge coordinator	"Integrator"	Component that runs three-way merge and mediates conflicts
False approval	"Reviewer hallucination"	Reviewer approves a diff with known bugs