BuildPython

Reviewer Agent: Separate Builder from Marker

The agent that wrote the code can't grade it. A reviewer is a second loop with its own system prompt, its own goal, and read-only access to everything the builder produced. Most of your reliability lives in the gap between the two.

TIME: ~55 minutesPREREQUISITES: Phase 14 · 38 (Verification Gate)

Learning Objectives

State why the same agent cannot reliably review its own work.
Build a reviewer agent loop that consumes builder artifacts and emits a structured review report.
Author a reviewer rubric that grades specific dimensions, not vibes.
Wire the reviewer into the workbench so the human review step starts from a real artifact.

The Problem

You ask the agent to fix a bug. It edits four files, runs the tests, and reports done. The verification gate (Phase 14 · 38) confirms acceptance ran and scope held, and says passed: true. You merge. Two days later you find the fix solved the wrong half of the bug.

Acceptance is necessary, not sufficient. A reviewer asks what acceptance can't: did this solve the right problem? Did it expand scope without flagging it? Did it document assumptions that should have been questioned? Did it leave the workbench in a state the next session can pick up?

Warm-up0/3

1. A verification gate reports passed: true: acceptance ran, tests exited 0, and scope held. What can you safely conclude?Warm-up

2. Why is an author usually a weak reviewer of their own work?Warm-up

3. Which question about an agent's change can a deterministic script answer with no judgment?Warm-up

Check: The Problem0/1

1. Why can't the agent that wrote a fix reliably grade that fix itself?Explain

The Concept

Mermaid diagram

A reviewer agent is a second loop that reads what the builder produced and scores it against a rubric. It has a different system prompt, a different goal, and read-only access to the diff, the state, the feedback, and the verdict.

Reviewer rubric

Five dimensions, each scored 0 to 2.

Dimension	Question
Problem fit	Did the change solve the task as stated, not a nearby task?
Scope discipline	Were edits confined to the contract or was the contract grown deliberately?
Assumptions	Are all hidden assumptions written down somewhere reviewable?
Verification quality	Does the acceptance command actually prove the goal, or did it prove a weaker version?
Handoff readiness	Could the next session pick up cleanly from the current state?

Total out of 10. Below 7 is a soft fail. Below 5, or any single dimension scored 0, is a hard fail.

The reviewer is a separate role, not a separate model

You can run the reviewer on the same model as the builder. What matters is the role separation: different system prompt, different inputs, no write access to the diff. The change in posture is what changes the signal.

The reviewer cannot edit the diff

The reviewer reads the diff, the state, the feedback, and the verdict, then writes a report. It does not patch anything. If the report says "fix this," the next builder turn makes the fix and the reviewer goes back to reviewing. Mix the roles and the gap disappears.

Reviewer rubric versus verification gate

The gate (Phase 14 · 38) checks deterministic facts: did acceptance run, did the rules pass, did scope hold. The reviewer makes qualitative judgments: was this the right work, is it documented, is the handoff usable. You need both.

Check: The Concept0/13

1. What does role separation require?Define

2. What is a reviewer rubric?Define

3. What is a soft fail verdict?Define

4. What triggers a hard fail?Define

5. What is a confidence floor on a review report?Define

6. Which artifacts does the reviewer agent read before it writes its report?Define

7. How does the lesson distinguish the reviewer rubric from the verification gate?Explain

8. Why does the reviewer get read-only access to the diff instead of being allowed to patch it?Explain

9. Why does the rubric score named dimensions such as problem fit and scope discipline instead of giving one overall impression?Explain

10. A change edits the right function, and the gate says passed: true. But the acceptance command only checks that the module imports, while the goal was correct output on malformed input. Which dimension should flag this?Analyze

11. A team lets its reviewer apply small patches to the diff itself. After a month its reports rarely catch real problems. Which cause fits best?Analyze

12. You want something to confirm the acceptance tests ran and no forbidden files were written. Where should those checks live?Design

13. The reviewer cannot judge the assumptions dimension because the state file is nearly empty. Which behavior fits a confidence floor?Design

Build It

code/main.py implements:

A ReviewerInputs dataclass that bundles the artifacts the reviewer reads.
A rubric scorer with one function per dimension. Each function is deterministic and stub-grade for the lesson; a real implementation would call an LLM.
A review_report.json writer holding the five scores, the total, and a verdict (pass, soft_fail, hard_fail).
Two demo cases: a clean change, and a "right tests, wrong problem" change.

Run it:

python3 code/main.py

You get two review reports written to disk and a console table of dimensional scores.

Check: Build It0/5

1. A reviewer scores a change 2, 2, 2, 2, 0 across the five dimensions, a total of 8. What verdict does the lesson's code return?Apply

2. In the code, the clean case has goal "add input validation to signup" and touches app/signup.py and tests/test_signup.py. What does score_problem_fit return?Apply

3. In the "wrong" demo case, only docs/api.md is touched, there is one scope.off_scope warning, no assumptions are recorded, task T-002 is still active, and pytest exited 0. What does the code print?Apply

4. score_handoff receives a state with active_task_id set to "T-007" and next_action set to "start T-008". What does it return?Apply

5. score_verification receives feedback records with exit codes 0, 1 and 0. What does it return?Apply

Production patterns in the wild

The receipts: Cloudflare's April 2026 AI Code Review system ran 131,246 review runs across 48,095 merge requests in 5,169 repos in 30 days. Median review time was 3 minutes 39 seconds. Up to seven specialist reviewers (security, performance, code quality, docs, release management, compliance, Engineering Codex) ran in parallel under a Review Coordinator that deduplicated findings and judged severity. The top-tier model was reserved exclusively for the coordinator, and the specialists ran on cheaper tiers.

Four patterns make this work at scale.

Specialist pool, not one big reviewer. One reviewer with a 5-dimension rubric works for a solo repo. Once the codebase has security-critical, performance-critical, and docs surfaces, split into specialists with smaller prompts. The coordinator handles deduplication, and the specialists never run the full rubric. Model-tier separation falls out of this: cheap specialists, expensive coordinator.

Bias mitigation is a design requirement, not an optimization. LLM judges show four reliable biases (Adnan Masood, April 2026). Position bias: GPT-4 is ~40% inconsistent on (A,B) versus (B,A) ordering. Verbosity bias: ~15% score inflation toward longer outputs. Self-preference: judges prefer outputs from their own model family. Authority: judges over-rate references to known authors. The mitigations are to evaluate both orderings and count only consistent wins, use 1-4 scales that explicitly reward conciseness, rotate judges across model families, and strip author names before scoring.

A calibration set, not vibes. Keep a 10-20 task historical set with known correct verdicts, and run the reviewer over it on every prompt change. If agreement with the historical record falls below 80%, revise the rubric before the reviewer ships. This is what every team eventually rediscovers, so start with it.

A hybrid norm with the gate. The verification gate (Phase 14 · 38) handles the deterministic checks: did acceptance run, did tests pass, did scope hold. The reviewer handles the semantic ones: was this the right work, are assumptions documented, is the handoff usable. Anthropic's 2026 guidance is explicit on this split: don't ask the reviewer to redo what the gate already proves.

Check: Production patterns in the wild0/7

1. What is a calibration set?Define

2. Which of these is NOT one of the four LLM-judge biases the lesson lists?Define

3. What does Cloudflare's 2026 review architecture look like?Explain

4. Which mitigations does the lesson pair with the four LLM-judge biases?Explain

5. You change the reviewer prompt, rerun it over your 15-task calibration set, and it agrees with the known verdicts on 10 tasks. What do you do?Apply

6. A reviewer prefers long, padded reports and ranks a pair of outputs differently when their order is swapped. Which mitigations address both problems?Analyze

7. A repo has security-critical, performance-critical and docs surfaces, and the single reviewer prompt has grown long and unfocused. Which design fits?Design

Use It

Three ways this shows up in production:

Claude Code subagents. A reviewer subagent runs after the builder closes a task and posts a comment on the PR with the rubric scores.
OpenAI Agents SDK handoffs. The builder hands off to a reviewer on task completion. The reviewer can hand back with a list of findings, or hand up to a human.
Two-model pairing. The builder runs on a faster, cheaper model. The reviewer runs on a stronger model with a smaller context, focused on judgment.

The reviewer is the second pair of eyes the workbench grows when humans can't do every review themselves.

Check: Use It0/1

1. Which production integration patterns does the lesson name for the reviewer agent?Explain

Ship It

outputs/skill-reviewer-agent.md generates a project-specific reviewer rubric, a reviewer agent stub wired to the builder's artifacts, and an integration with the verification gate. Human review then starts from a written report instead of a blank page.

Check: Ship It0/1

1. Why is the reviewer's report wired into the workbench ahead of the human review step?Explain

Author's Note

I'd build the calibration set before I tuned a single line of the rubric. The lesson sets the bar plainly: a 10-20 task historical set with known verdicts, rerun on every prompt change, and agreement below 80% means the rubric gets revised before the reviewer ships. Until a reviewer has cleared that bar, I treat its scores as unverified. I also treat its report as a starting page for the human reviewer, and not as a replacement for the gate, which already proves the deterministic facts. My bet is that a reviewer with a checked rubric and a narrow job earns trust, while a clever prompt on its own doesn't.

Exercises

Add a sixth dimension specific to your product domain. Defend why it is not absorbed by the existing five.
Run the reviewer with two different system prompts (terse, verbose). Which produces a report a human is more likely to read?
Add a confidence field per dimension. Refuse to ship the report when confidence in the lowest dimension is below 0.6.
Build a calibration set: 10 historical task close-outs with known correct verdicts. Run the reviewer over them. Where does it disagree with the historical record?
Add a "request more evidence" affordance: the reviewer can ask the builder for a specific test run before scoring. What is the right back-off so this does not loop?

Key Terms

Term	What people say	What it actually means
Reviewer rubric	"Checklist"	Five-dimension 0-2 scoring with a written question per dimension
Soft fail	"Needs revisions"	Total below 7 (but not a hard fail); builder gets findings to address
Hard fail	"Reject"	Total below 5 or any dimension at 0; halt and surface to human
Role separation	"Different prompt"	Same model can be both roles; the discipline is inputs and posture
Confidence floor	"Don't ship low-signal reports"	Refuse to emit a verdict when the rubric is uncertain