BuildPython

Role Specialization — Planner, Critic, Executor, Verifier

The most common multi-agent decomposition in 2026: one agent plans, one executes, one critiques or verifies. MetaGPT (arXiv:2308.00352) formalizes this as SOPs encoded into role prompts (Product Manager, Architect, Project Manager, Engineer, QA Engineer), following `Code = SOP(Team)`. ChatDev (arXiv:2307.07924) chains designer, programmer, reviewer, tester through a "chat chain" with "communicative dehallucination" (agents explicitly request missing details). The verifier is load-bearing: Cemri et al. (MAST, arXiv:2503.13657) found verification gaps behind 21.3% of traced failures, and many of the rest trace back to a check that failed silently or never ran. PwC reported a 7× accuracy gain (10% → 70%) from structured validation loops in CrewAI.

TIME: ~60 minutesPREREQUISITES: Phase 16 · 04 (Primitive Model), Phase 16 · 05 (Supervisor)

Learning Objectives

Name the four canonical roles (planner, executor, critic, verifier) and the tools and output each one gets.
Explain how MetaGPT's SOP pattern and ChatDev's communicative dehallucination make role prompts predictable.
Distinguish a subjective critic from a deterministic verifier, and say why a pipeline needs at least one verifier decided by code.
Run the four-role toy pipeline and read why the critic approves buggy code while the verifier rejects it.
Apply the Ship It checklist (critic before verifier, a 2-round revision budget) to a role roster for a new task.

Problem

Generic multi-agent systems produce generic output. Three coders in a group chat write three flavors of the same mediocre code. You can add more agents, add more rounds, and still not cross the quality threshold.

The fix is different agents, not more of them. Assign distinct roles. Give the critic tools the planner does not have. Give the verifier an objective test suite. Now the system has internal disagreement with grounded correction, not just parallel guessing.

Warm-up0/3

1. Which four primitives make up the multi-agent primitive model?Warm-up

Three agent boxes in a row, connected by arrows labeled message.

2. In the supervisor / orchestrator-worker pattern, what does the lead agent do?Warm-up

A lead agent box above two worker boxes, with arrows for a sub-task down and a result back up, and a dashed box asking what the lead's job is.

3. A teammate says a function "looks right" after reading it. Which check gives the strongest evidence that it works?Warm-up

Check: Problem0/2

1. What is role specialization in a multi-agent system?Define

A pipeline of four named role boxes in sequence: Planner, Executor, Critic, Verifier.

2. Three coder agents in a group chat keep producing the same mediocre code. What fixes this?Explain

Three coder boxes all feeding into one box labeled mediocre code times three, with a dashed box asking what fixes this.

Concept

The four canonical roles

Planner. Reads the goal, produces a step list or a spec. Tools: knowledge retrieval, docs. Output: structured plan.

Executor. Reads one plan step at a time, produces the artifact. Tools: the actual work tools (code compiler, shell, API client). Output: the artifact.

Critic. Reads the executor's output against the planner's intent. Tools: read-only access to the artifact, static analysis. Output: accept/reject with reasons.

Verifier. Reads the artifact and runs a deterministic check. Tools: test runner, type checker, schema validator. Output: pass/fail with evidence.

Critic is subjective, opinionated, often LLM-based. Verifier is objective, deterministic, often code-based. They are different roles.

MetaGPT's SOP pattern

MetaGPT (arXiv:2308.00352) encodes software engineering SOPs as role prompts:

Product Manager writes the PRD.
Architect produces the system design.
Project Manager splits tasks.
Engineer implements.
QA Engineer runs tests.

Each role has a strict input/output schema. The role prompt says what the role is and what it must produce. The Code = SOP(Team) formulation says that deterministic SOPs turn a team of LLMs into a predictable pipeline.

ChatDev's communicative dehallucination

ChatDev adds a key move: when an executor needs a specific detail that was not in the plan, it explicitly asks the designer before continuing. This prevents the classic LLM failure of plausibly inventing the detail.

Implementation: the role prompt includes "when you need specific information you were not given, ask the relevant role by name before producing output."

Why verifier matters most

Cemri et al. (MAST) traced 1642 multi-agent execution failures. 21.3% were verification gaps: the system shipped an answer no one had checked. The remaining 79% often trace back to a check that failed silently or was never run. Verification is the load-bearing role.

PwC reported (CrewAI deployments, 2025) that adding a structured validation loop moved accuracy from 10% to 70%. That is a 7× gain from adding one role.

Critic vs verifier

A critic is an LLM reviewing an artifact for quality. Subjective. Can be fooled by plausible prose.
A verifier is a deterministic program running on the artifact. Objective. Gives pass/fail with evidence.

Use both. Critic catches taste issues the verifier cannot articulate. Verifier catches bugs the critic cannot see because they show up only at runtime.

The anti-pattern

Every role in your system is an LLM, and each one's output is "looks good to me." This is the classic MAST failure mode. Add at least one verifier whose pass/fail is decided by code, not by an LLM.

Framework mappings

CrewAI: Agent(role, goal, backstory) is the textbook specialization surface.
LangGraph: nodes can have specialized prompts; edges enforce the pipeline.
AutoGen: role-specific ConversableAgents with one-word names in a GroupChat.
OpenAI Agents SDK: handoff tools between role-specialized Agents.

Check: Concept0/19

1. What is communicative dehallucination, as ChatDev introduces it?Define

A plan missing an output format, next to an executor with two dashed options, asking the planner or guessing, each leading to an unresolved result box.

2. What does MetaGPT's formulation Code = SOP(Team) mean?Define

The formula Code = SOP(Team) above a row of four generic role boxes connected by arrows, with a note that this is a team of role-prompted agents chained in order.

3. What is a critic in a role-specialized pipeline?Define

The four-role pipeline with the Critic box highlighted and a dashed box asking what kind of role it is.

4. What is a verifier in a role-specialized pipeline?Define

The four-role pipeline with the Verifier box highlighted and a dashed box asking what kind of role it is.

5. What is a verification gap?Define

A bar chart of 1,642 traced multi-agent failures, with a verification gap slice at 21.3 percent next to other causes at 78.7 percent.

6. Which design is the anti-pattern to avoid when building a role roster?Define

7. MetaGPT gives each role a strict input/output schema modeled on a software team. Which roster does it use?Define

A row of five generic role boxes labeled Role 1 through Role 5, each tagged with an input/output schema.

8. PwC reported accuracy rising from 10% to 70% in CrewAI deployments. What change produced the gain?Define

A bar chart of CrewAI deployment accuracy going from 10 percent before to 70 percent after, with a dashed box asking what changed.

9. Which tool access fits the critic role?Explain

The four-role pipeline with the Critic box highlighted and a dashed box asking what tools it gets.

10. Why does communicative dehallucination reduce invented details?Explain

An executor missing a retry policy detail, with two dashed options, asking the planner or inventing a guess, each leading to an unresolved outcome box.

11. How do strict per-role input/output schemas make a pipeline predictable?Explain

The four-role pipeline with a schema tag on every box, and a dashed box asking why this makes the pipeline predictable.

12. What distinguishes a critic from a verifier?Explain

The critic and verifier boxes both highlighted in the pipeline, each with a dashed box asking its nature.

13. Why use both a critic and a verifier instead of only one of them?Explain

The critic and verifier boxes both highlighted, each with a dashed box asking what it catches.

14. How do AutoGen and the OpenAI Agents SDK each specialize roles?Explain

15. How do CrewAI and LangGraph each specialize roles?Explain

16. A pipeline extracts invoice fields into JSON. Which check qualifies as its verifier under the lesson's definition?Apply

An invoice PDF feeding an LLM extractor, with a table of illustrative output fields and types, and a dashed box asking what the verifier is here. — Illustrative invoice fields; exact names are not given in the lesson.

17. A planner, a coder and a reviewer are all LLM agents, and the pipeline still ships a function that crashes on empty input. Which change addresses the root cause?Analyze

A planner, a coder and a reviewer, all marked LLM, chained to a shipped box that crashes on empty input, with a dashed box asking which change fixes the root cause.

18. A critic approves buggy code. The team swaps in a stronger LLM critic and the bug still ships. Which assumption is wrong?Analyze

Two attempts with the same executor, critic approving each time with a stronger version in attempt two, both shipping the same bug, and a dashed box asking what assumption is wrong.

19. An executor needs a base URL the plan never mentioned, and the shipped code contains a plausible but wrong one. Which practice would have prevented the invented value?Analyze

A plan that never mentions a base_url, next to executor output containing a plausible but wrong URL, with a dashed box asking what would have prevented this.

Build It

code/main.py implements a 4-role pipeline for a simple Python function, add_two:

Planner returns a hardcoded spec: a signature and three test cases.
Executor returns a code string. There are two versions, one correct and one buggy.
Critic is a string match standing in for an LLM reviewer. It checks that def, return, and the function name from the spec appear in the code.
Verifier runs the generated code with exec in a fresh namespace and calls the function on each test case.

The demo runs twice. In the first run the executor writes a + b, and critic and verifier both pass. In the second it writes a * b. The critic approves it because all three of its checks pass on plausible-looking code, and the verifier catches it because all three tests fail.

Run:

python3 code/main.py

Check: Build It0/3

1. In the Build It demo's second run, why does the critic approve code that the verifier rejects?Apply

A terminal showing the critic approving the buggy executor's code while the verifier reports three failing test calls, with a dashed box asking why the verdicts split.

2. In the toy pipeline, the executor returns code containing def and add_two but no return statement. What does the string-match critic do?Apply

A hypothetical executor output with def and the function name but no return statement, next to a critic checklist with each check left as a question mark.

3. In the first demo run the executor writes a + b. What do the critic and the verifier return?Apply

Use It

outputs/skill-role-designer.md takes a task and produces the role roster (3-5 roles), the input/output schema per role, and the verifier check. Use this before wiring agents into a framework.

Ship It

Checklist:

At least one deterministic verifier. Don't build the whole pipeline from LLM roles.
Explicit I/O schema per role. The planner returns a spec, not prose, and the executor reads that schema.
Communicative dehallucination. The executor asks the planner when info is missing instead of inventing it.
Critic/verifier ordering. Run the critic first (cheap, catches design issues), the verifier second (slow, catches bugs).
Loop budget. Max 2 critic-executor revision rounds before escalating to a human.

Check: Ship It0/6

1. What is a revision loop?Define

2. What loop budget does the Ship It checklist set for critic-executor revision rounds?Define

A critic and executor box joined by reject and retry arrows in a loop, with a human box off to the side labeled after an unstated number of rounds, and a dashed box asking the round cap.

3. What order does the Ship It checklist recommend for the critic and the verifier?Explain

Two candidate orderings of a critic box and a verifier box, one above the other, with a dashed box asking which runs first.

4. A critic-executor loop has no cap. The critic keeps rejecting small wording issues and a run never finishes. What was left out of the design?Analyze

A critic rejecting the same executor over and over with a round counter reading many and no end in sight, with a dashed box asking what was left out of the design.

5. You must build a pipeline that generates database migrations. Which roster follows the lesson's guidance?Design

A task labeled generate database migrations above four empty role slots, with a dashed box asking the roster.

6. A new task needs low latency but must still catch bugs. Which design fits the Ship It checklist?Design

Author's Note

I'd run main.py and read the second run before anything else. The critic approves a * b because its only checks are that def, return, and add_two appear somewhere in the string. The verifier calls the function on the three spec tests, and all three fail, with the expected and actual values printed for each. That output is the argument for a check decided by code, in about 130 lines.

I'd also keep the toy in proportion. The planner returns a hardcoded spec and the executors return fixed strings, so nothing here exercises communicative dehallucination or the 2-round revision budget from Ship It. The pipeline runs the critic and the verifier on each artifact instead of gating the verifier behind the critic, which is an easy change to try. And exec in a fresh dict is not isolation. For code a model actually wrote, I'd run the verifier in a subprocess with a timeout.

Exercises

Run code/main.py and observe how the verifier catches the bug the critic missed. Add a static-analysis check (count occurrences of return) as an additional verifier. What does it catch that the runtime test misses?
Add a 5th role: "requirements analyst" that translates user wish into planner-ready spec. What communicative dehallucination requests should flow up to it?
Read MetaGPT Section 3 ("Agents"). List the input/output schema of each of MetaGPT's 5 roles.
Read ChatDev's chat-chain diagram (arXiv:2307.07924 Figure 3). Identify where communicative dehallucination breaks a loop that would otherwise be infinite.
PwC's 7× accuracy gain came from verification loops. Hypothesize three tasks where adding a verifier would not help, because deterministic checking of correctness is impossible or prohibitively expensive.

Key Terms

Term	What people say	What it actually means
Role specialization	"Different agents, different jobs"	Distinct system prompts tuned for planner/executor/critic/verifier roles.
SOP pattern	"Encoded standard operating procedure"	MetaGPT's framing: strict I/O schemas per role turn a team into a pipeline.
Communicative dehallucination	"Ask before inventing"	ChatDev pattern: executor asks planner when a detail is missing rather than making one up.
Critic	"LLM reviewer"	Subjective, opinionated reviewer. Catches taste issues. Can be fooled by plausible prose.
Verifier	"Deterministic check"	Code-based pass/fail. Test runner, type checker, schema validator. Cannot be fooled.
Verification gap	"No one checked"	21.3% of MAST failures. Answer shipped without a check that would have caught the bug.
Revision loop	"Critic sends it back"	Critic rejection triggers executor re-run with feedback. Needs a budget.
All-LLM anti-pattern	"Looks good to me"	Every role is an LLM, no deterministic check. Classic MAST failure.