BuildPython

Multi-Agent Debate and Collaboration

Du et al. (ICML 2024, "Society of Minds") run N model instances that propose answers independently, then critique each other over R rounds, converging on an answer. It improves factuality, rule-following, and reasoning. A sparse topology beats a full mesh on token cost.

TIME: ~60 minutesPREREQUISITES: Phase 14 · 12 (Workflow Patterns), Phase 14 · 05 (Self-Refine and CRITIC)

Learning Objectives

Explain the debate protocol: N proposers, R rounds, converge on a shared answer.
Describe why debate improves factuality, rule-following, and reasoning.
Explain sparse topology: not every debater needs to see every other.
Implement a stdlib debate over a scripted LLM with full-mesh and sparse variants; measure token cost vs accuracy.

The Problem

Self-Refine (Lesson 05) has one model critique itself, which risks groupthink: the critic shares every blind spot of the author. CRITIC (Lesson 05) fixes that by grounding the critique in external tools, but you don't always have a tool that can check the answer. Debate is the third option. Several instances answer independently, critique each other, and settle by working through their disagreements.

Warm-up0/3

1. Why is it a weakness when a model critiques its own draft in a Self-Refine loop?Warm-up

2. In Anthropic's orchestrator-workers workflow, what does the orchestrator do?Warm-up

3. A pipeline makes 5 LLM calls per question instead of 1. What mainly grows as a result?Warm-up

Check: The Problem0/1

1. Self-Refine's critic shares the author's blind spots, and CRITIC needs a tool that can check the answer. When does debate fit?Explain

The Concept

Multi-agent debate is a protocol where N model instances each propose an answer to the same question, then spend R rounds reading and critiquing one another's answers before returning a shared one. The disagreement between instances is the signal. Where they agree, you gain confidence. Where they split, the critique rounds force someone to defend or drop a claim.

Society of Minds (Du et al., ICML 2024)

N model instances independently propose answers to the same question.
Over R rounds, each model reads the others' proposals and critiques them.
Models update their answers based on the critiques.
After R rounds, return the convergent answer.

The original experiments used N=3, R=2 because of cost. On hard problems (MMLU, GSM8K, Chess Move Validity, biography generation), accuracy keeps improving as you add agents and rounds.

Mixing models helps too. Cross-model combinations beat single-model debates: in the paper's setup, ChatGPT + Bard together outperformed either one alone.

Sparse topology

"Improving Multi-Agent Debate with Sparse Communication Topology" (arXiv:2406.11776, 2024-2025) showed that full-mesh debate is not always the best layout. Sparse topologies (star, ring, hub-and-spoke) can match its accuracy at lower token cost, because each debater sees only a subset of its peers.

The savings are easy to count:

Full mesh, N=5, R=3: 5 × 3 = 15 proposals, each reading 4 peers, which is 60 critique ops.
Star, N=5, R=3 (one hub + 4 spokes): the same 15 proposals, but spokes read only the hub, which is 12 critique ops.

When debate helps

Factuality. N independent proposals cross-check each other, which reduces hallucination.
Rule-following. In chess move validity, one model misses a rule and the others catch it.
Open-ended reasoning. Several framings of the problem narrow in on the right answer.

When debate hurts

Latency-sensitive UX. N × R serial rounds is latency you may not have.
Cost-sensitive scale. You pay N × R tokens for every question.
Simple factual lookups. One lookup is cheaper than five debates.

2026 practical instantiations

Anthropic orchestrator-workers (Lesson 12) is one variant of debate, with a synthesis step at the end.
LangGraph supervisor (Lesson 13): a central router plus specialist agents can implement debate as a node.
OpenAI Agents SDK (Lesson 16): agents hand off back and forth for iterative critique.
Multi-agent evals: pair debate with an evaluator-optimizer to get an eval signal.

Where this pattern goes wrong

Convergence collapse. All agents converge on the first wrong answer. Mitigate with required disagreement rounds.
Hub failure. In a star topology, a bad hub corrupts everyone who reads it. Rotate the hub or use multiple hubs.
Prompt homogenization. If every agent gets the same prompt, they produce the same answers and there is nothing to critique. Use diverse prompts, diverse models, or both.

Check: The Concept0/27

1. What is multi-agent debate?Define

2. In a full-mesh debate, who reads whom each round?Define

3. What defines a sparse debate topology?Define

4. What is a hub-and-spoke (star) debate topology?Define

5. In a debate, what does convergence mean?Define

6. What is Society of Minds?Define

7. What is the main token-cost win of a sparse topology over full mesh?Explain

8. What is convergence collapse?Explain

9. On hard problems such as MMLU and GSM8K, how does debate accuracy behave as you add agents and rounds?Explain

10. The original Society of Minds experiments used only N=3 and R=2. Why?Explain

11. Why is disagreement between debaters useful as a signal?Explain

12. Why does debate reduce hallucination on factual questions?Explain

13. In chess move validity, why does debate help with rule-following?Explain

14. Which case does the lesson list as one where debate hurts?Explain

15. How did cross-model debate compare with single-model debate in the Society of Minds setup?Explain

16. Which production pattern does the lesson call a variant of debate?Explain

17. How can a LangGraph supervisor implement debate?Explain

18. What is the 'hub failure' risk in a star (hub-and-spoke) debate topology?Explain

19. For N=5, R=3, what critique-op counts does the lesson compute for full mesh versus star topology?Apply

20. A full-mesh debate uses N=4 and R=2, and each proposal reads its 3 peers. How many critique ops does it run?Apply

21. A star debate has N=6 and R=2, with one hub and five spokes that read only the hub. How many proposals and critique ops does it run?Apply

22. Five debaters share one prompt and keep returning identical answers, so critique adds nothing. What is the fix?Analyze

23. Every debater returns the same wrong answer in round 1, and later rounds only restate it. Which missing safeguard explains this?Analyze

24. A team cuts debate cost by lowering N from 5 to 2, and accuracy drops. What did the design choice get wrong?Analyze

25. Your five-debater, three-round setup costs too much, but you want to keep all 15 proposals. Which change fits?Design

26. Your team ships a latency-sensitive support bot that answers simple FAQ lookups. Should you run N x R debate on it?Design

27. You must check answers that no external tool can verify, under a tight cost budget at N=5, R=3. Which design fits?Design

Build It

code/main.py implements a stdlib debate:

Debater class: a scripted LLM with per-debater opinion drift.
FullMeshDebate and SparseDebate runners.
Three questions: one factual, one rule-based, one reasoning.
Metrics: convergent answer, rounds to convergence, total critique ops.

Run it:

python3 code/main.py

The output gives per-protocol accuracy and cost. Sparse matches full mesh on 2/3 questions at lower cost.

Check: Build It0/1

1. You run main.py, which scores full-mesh and sparse debates on three questions. The lesson says sparse matches full mesh on 2 of the 3. What should the cost columns show?Apply

Use It

Anthropic orchestrator-workers fit simple debates with 2-3 workers.
LangGraph fits stateful multi-round debate with checkpointing.
Custom code fits research or specialized correctness guarantees.

Check: Use It0/1

1. You need a multi-round debate that keeps its state and can resume from a checkpoint after a crash. Which option fits best?Design

Ship It

outputs/skill-debate.md scaffolds a multi-agent debate with configurable topology, N, R, and a convergence rule.

Author's Note

I'd reach for a sparse star topology before cutting N or R. The star keeps all 15 proposals at N=5, R=3 and drops critique ops from 60 to 12, so it cuts the cost sharply without shrinking the pool of proposals. The price is the hub, which every spoke reads, so I'd want to rotate it or run more than one. I'd also check that the debaters can actually disagree before paying N × R at all. With one shared prompt they hand back the same answer five times, and no topology fixes that. Vary the prompts or the models first, then tune the layout.

Exercises

Implement a "forced disagreement" rule: in round 1, every debater must produce a distinct proposal. Measure effect on convergence speed.
Add a confidence-weighted aggregation: debaters return (answer, confidence); aggregator weights by confidence. Does it help?
Swap one "agent" for a different scripted LLM with different opinions. Does heterogeneity improve accuracy?
Measure token cost for full mesh vs sparse on your 3 questions. Plot cost vs accuracy.
Read the Society of Minds paper. Port your toy to N=5, R=3. What breaks? What gets better?

Key Terms

Term	What people say	What it actually means
Debate	"Multi-agent critique"	N proposers, R rounds of cross-critique, converge
Full mesh	"Everyone reads everyone"	Every debater reads every peer each round
Sparse topology	"Limited peer view"	Debaters read only a subset of peers
Hub-and-spoke	"Star topology"	One central debater, N-1 spokes read only the hub
Convergence	"Agreement"	Debaters converge on a shared answer
Society of Minds	"Du et al. debate paper"	ICML 2024 multi-agent debate method