Capstone Lesson 25: Verification Gates and the Observation Budget

A verification gate is the harness component that says no. It is a deterministic function of (call, history, ledger) that returns either ALLOW or DENY with a reason. It is not a model and not a judge. The reason is logged, the model is told, and the loop continues or aborts.

You need one because a harness that lets the model call tools freely runs into three classes of bug within the first hour of real use.

The first is unbounded observation. A grep across a 200K-line repo dumps half a million tokens into the next turn. The model gets one match per kilobyte, the rest of the context is wasted, the token bill is large, and the agent ends up worse at the task, not better.

The second is stale recency. A long-running task piles up fifty tool calls, and the model rereads the first read_file from turn three as if it were live state. Edits made on turn forty-seven never show up, because the prompt builder serialized the earliest observations first.

The third is privilege creep. A research task starts with web_search and somehow ends up running shell, because the model invented a tool name and the harness defaulted to permissive. By the time anyone reads the trace, a junk file is sitting in /tmp and a curl has hit a private API.

A gate is anything with an evaluate(call, ctx) -> GateDecision method. The chain is an ordered list of gates, and evaluation short-circuits on the first deny. Order matters: cheap structural gates run before expensive token-counting ones.

This lesson ships four gates:

• WhitelistGate holds an explicit set of allowed tool names and denies anything outside it. It is the cheapest gate, so it runs first.
• RegexGate matches tool arguments against a regex. Use it to refuse shell calls containing rm -rf, or HTTP calls to internal IPs. It is pure on the call payload.
• RecencyGate refuses a call when the last recorded observation is more than N turns old, which forces a fresh read instead of relying on stale state. The first call in a session always passes.
• BudgetGate enforces a ceiling on the cumulative tokens the model has read across the session. Once the ledger says the ceiling is reached, every further tool call is denied.

The observation ledger is the bookkeeping behind them. Every successful tool call writes one row: turn, tool name, the text shown, and its token count. The ledger answers two questions, how much the model has seen in total and how much it has seen of tool X. The budget gate reads the first. A per-tool budget gate, which you will write as an exercise, reads the second.

The harness asks the chain, and the chain either nods or refuses. On a nod, the tool runs, the ledger ticks, and the result is appended to the message store. On a refusal, the model gets the refusal as a system message and the loop decides whether to retry or abort.

The implementation is a single main.py plus tests.

1. Observation and ToolCall dataclasses define the wire shapes.
2. ObservationLedger records (turn, tool, tokens) rows and answers cumulative() and per_tool(name).
3. GateDecision carries (allow, gate, reason).
4. VerificationGate is the protocol. Each gate implements evaluate(call, ctx).
5. GateChain wraps an ordered list. It calls each gate and returns the first deny, or allow if every gate passes.
6. The demo runs five synthetic calls through a 200-token budget: three are allowed and two are refused. The budget gate refuses the fourth call (322/200 tokens), and the whitelist refuses the fifth (shell), so the loop reports a clean refusal count of 2.

The token counter is deliberately crude: len(text) // 4. This lesson is about the gate plumbing, not the tokenizer, so swap in a real tokenizer in production.

A deny is cheaper than an allow. WhitelistGate is an O(1) hash lookup. RegexGate costs O(pattern * argv). RecencyGate reads one value from the ledger (latest_turn), and BudgetGate reads the entire ledger. Ordering the gates by ascending cost lets a denied call short-circuit before any expensive work happens.

Blast radius points the same way. The whitelist makes the strongest claim: this tool is not in the contract. The regex gate comes next: this argument is not in the contract. Recency follows, because the harness still cares but the call is structurally legal. Budget goes last since, by definition, it only fires when everything else has passed.

The previous lessons gave you the loop, the tool registry, the message store, the prompt builder, and the model router. This lesson adds the layer between the model and the tools.

Lesson 26 ships the sandbox the dispatcher hands the tool call to once the chain says ALLOW. Lesson 27 ships the eval harness, which records refusal counts as a quality signal. Lesson 28 wires the gate decisions into OpenTelemetry spans, and Lesson 29 stitches everything into a working coding agent.

cd phases/19-capstone-projects/25-verification-gates-observation-budget
python3 code/main.py
python3 -m pytest code/tests/ -v

The demo prints a turn-by-turn trace that includes every gate decision, and exits zero. The tests cover the ledger, each gate in isolation, the chain short-circuit, and the synthetic loop end to end.