BuildPython

Verification Gates

The agent doesn't get to mark its own work as done. A verification gate reads the scope contract, the feedback log, the rule report, and the diff, then answers one question: is this task actually complete? If the gate says no, the task isn't done, whatever the chat says.

TIME: ~55 minutesPREREQUISITES: Phase 14 · 33 (Rules), Phase 14 · 36 (Scope), Phase 14 · 37 (Feedback)

Learning Objectives

Define a verification gate as a deterministic function over workbench artifacts.
Combine rule report, scope report, feedback records, and diff into a single verdict.
Emit a `verification_report.json` the reviewer agent and CI can both read.
Refuse to advance a task on any block-severity failure, without exception.

The Problem

Agents declare success too easily. Three failure shapes dominate:

"Looks good." The model read its own diff and decided it was correct.
"Tests passed." Said with confidence, with no record of the tests actually running.
"Acceptance met." The acceptance criteria got interpreted loosely enough to mean "anything resembling done."

The workbench fix is a single verification gate: a function that reads the artifacts the agent already produced and makes the call. It is deterministic, it lives in version control, and it is wired into CI. The agent can't bribe it.

Warm-up0/3

1. An agent ends a task by saying 'tests passed.' Which evidence is the most trustworthy?Warm-up

2. Why should a check that gates a merge return the same result on the same inputs?Warm-up

3. A scope contract lists the files an agent may edit. The agent changes a file that isn't on the list. What does that mean?Warm-up

Check: The Problem0/1

1. An agent says 'tests passed,' but the feedback records contain no entry for the acceptance command. Why must the gate fail the task?Explain

The Concept

A verification gate is a deterministic function over workbench artifacts that produces a pass/fail verdict for a task. It reads the scope report, the rule report, the feedback records, and the diff, and it answers one question: is this task actually complete?

Mermaid diagram

What the gate checks

Check	Source artifact	Severity
All acceptance commands ran	`feedback_record.jsonl`	block
All acceptance commands exited zero	`feedback_record.jsonl`	block
Scope check has no forbidden writes	`scope_report.json`	block
Scope check has no off-scope writes	`scope_report.json`	warn (block under `--strict`)
All block-severity rules pass	`rule_report.json`	block
No `null` exit codes in feedback	`feedback_record.jsonl`	block
Touched files match `scope.allowed_files`	both	warn

A warn finding annotates the verdict. A block finding prevents passed: true.

Deterministic, not probabilistic

The same artifact set must produce the same verdict every time, so there are no LLM judges inside the gate. They belong on the reviewer side (Phase 14 · 39), where the goal is qualitative evaluation rather than status.

One report, one path

The gate writes one verification_report.json per task close-out, under outputs/verification/<task_id>.json, and CI reads the same path. Multiple gates with different paths fork the source of truth.

Refuse without exception

The agent can't override a block-severity finding. Only a human can, with a recorded override_reason and an overridden_by user id. The override is a signed change, not an agent decision.

Check: The Concept0/12

1. What is a verification gate?Define

2. What does a block-severity finding do to a task?Define

3. What is an acceptance command?Define

4. What does the one-report-path rule say?Define

5. Who can override a block-severity finding, and how?Explain

6. Both forbidden writes and off-scope writes come from scope_report.json. How does the gate treat each when --strict is off?Explain

7. Why is a single report path better than several gates each writing their own file?Explain

8. Every acceptance command exited zero, but scope_report.json lists a forbidden write. What does the gate return?Apply

9. A human reads a report showing passed: true for task T-9, but CI blocked the merge. The gate had written its report to a local folder, while CI reads outputs/verification/T-9.json. Which principle was broken?Analyze

10. After a team adds an LLM judge to the gate, one task passes on a CI run and fails on a rerun with identical artifacts. What went wrong, and what is the fix?Analyze

11. A product owner wants the pipeline to also reject code that is hard to read. How should you add this without weakening the gate?Design

12. During an incident, a task with a block finding must merge now. Which path keeps the gate trustworthy?Design

Build It

code/main.py implements:

A loader for each input artifact, all stubbed locally so the lesson is self-contained.
A verify(task_id, artifacts) -> VerdictReport pure function.
A printer that shows the per-check results and the final pass/fail.
A demo with three task scenarios: clean pass, scope creep, missing acceptance.

Run it:

python3 code/main.py

Output: three verdict reports, each saved next to the script as verification_report_<task_id>.json (the demo's stand-in for the outputs/verification/<task_id>.json path described above). The override demo appends to overrides.jsonl in the same folder.

Check: Build It0/6

1. Task T-002 in the demo has a forbidden write to scripts/release.sh, an off-scope write to README.md, a failed rule, and coverage of 0.62 against a previous 0.80 (default floor 0.80, no --strict). Which result does verify return?Apply

2. The feedback has one record for the only acceptance command, with exit_code None. Scope and rules are clean, and coverage is 0.85 against 0.85. What does verify produce?Apply

3. The only acceptance command exited 0, and a second logged command, 'ruff check .', exited 1 but is not an acceptance command. Scope, rules and coverage are clean. What does verify return?Apply

4. Coverage is 0.83 now and was 0.835 at the previous merge. The floor is the default 0.80, and everything else is clean. What does verify return?Apply

5. Everything is clean except that no coverage report is supplied. How do strict=False and strict=True differ?Apply

6. record_override is called with every field filled except reason, which is an empty string. What happens?Apply

Production patterns in the wild

Five patterns turn the gate from "another lint job" into the deciding edge.

Defense in depth, not a single gate. Pre-commit hook, CI status check, pre-tool authz hook, pre-merge gate. Each layer is deterministic, so a failure in one is caught by the next. microservices.io's March 2026 playbook is explicit that the pre-commit hook is non-bypassable because, unlike a model-side skill, it doesn't depend on the agent following instructions. The verification gate sits at the CI / pre-merge layer.

Deterministic checks for the verdict, model judges only for nuance. The lesson's description of Anthropic's 2026 Hybrid Norm pairs two kinds of signal. Verifiable rewards (unit tests, schema checks, exit codes) answer "did the code solve the problem?", and LLM rubrics answer "is the code readable, secure, on-style?" The gate runs the first class and the reviewer (Phase 14 · 39) runs the second. Mixing them collapses the signal.

Signed override log, not Slack threads. Every override emits a row in outputs/verification/overrides.jsonl with: timestamp, finding code, reason, signing user, current HEAD commit. The runtime refuses any override that lacks the signature, and the audit trail is git-tracked. That is the line between an override policy and override theater.

Coverage floor as a first-class check. A coverage_report.json feeds a coverage_floor (default 80%) check. The gate fails if measured coverage drops below the floor, or below the previous merge's floor by more than 1 percentage point. Without this check, agents quietly delete tests that fail and the verification reports stay green.

--strict mode promotes warns to blocks. Use it for release branches, ship-blocking PRs, or post-incident triage, where every warning should be a hard fail. Turn it on per branch rather than globally, because strict-on-everything corrodes day-to-day flow.

Check: Production patterns in the wild0/8

1. What is an override log?Define

2. Which work belongs in the gate, and which belongs on the reviewer side?Explain

3. How does defense in depth layer the gates?Explain

4. What does a coverage_floor check protect against?Explain

5. An override request arrives with no signing user. What should the runtime do?Explain

6. Over a week an agent's runs all show green acceptance commands and passing verdicts, yet test coverage fell from 90% to 71% and several test files are gone. Which missing check would have caught this?Analyze

7. Months later an auditor asks why a task with a failing block finding was merged. The only trace is a Slack thread that says 'ok to ship.' Which pattern was missing?Analyze

8. A team ships from a release branch and wants even small anomalies to stop the merge. What should they do?Design

Use It

Three places the gate runs:

CI step. A verify_agent job runs the gate against the agent's final artifacts. Merge protection refuses without passed: true.
Pre-handoff hook. The agent runtime calls the gate before generating the handoff doc. No green verdict, no handoff.
Manual triage. Operators read the report when an agent claims success and a human suspects it.

The gate is the deciding edge in the workbench flow. Every other surface is upstream of it.

Check: Use It0/1

1. Why does the agent runtime call the gate before it generates the handoff doc?Explain

Ship It

outputs/skill-verification-gate.md wires the gate into a specific project: which acceptance commands feed it, which rules are block-severity, which off-scope writes are tolerated, and how the override audit log is stored.

Author's Note

I'd keep the gate boring. My bet is that its verdict is only worth trusting while it stays a pure function of the artifacts on disk, so I treat any proposal to let a model weigh in on pass/fail as a proposal to move that work to the reviewer. Judgment about readability and style is useful, and Phase 14 · 39 is where it goes. The gate's job is to say whether the acceptance commands ran and exited zero, and it should say the same thing on every run.

Exercises

Add a coverage_floor check: the test command must produce a coverage report with at least 80%. Decide which artifact carries the floor.
Support a --strict mode that promotes every warn to block. Document the cases where strict mode is the right default.
Make the gate produce a Markdown summary in addition to JSON. Defend which fields belong in the summary.
Add a time_since_last_human_touch check: any file edited within 60 seconds of a human keystroke is exempt from off-scope flags.
Run the gate on a real agent diff from your product. How many findings are real and how many are noise? Where does the gate need to grow?

Key Terms

Term	What people say	What it actually means
Verification gate	"The check that stops things"	Deterministic function over workbench artifacts producing a pass/fail verdict
Block severity	"Hard fail"	A finding that prevents `passed: true` and requires a signed override
Override log	"Why we let it through"	Signed entries with reason and user id, audited by review
Acceptance command	"The proof"	A shell command whose zero exit is what `done` means
One report path	"Source of truth"	`outputs/verification/<task_id>.json`, consumed by CI and humans alike