Capstone Lesson 29: End-to-End Coding Agent on the Harness

Most agent demos work in isolation: a sandbox by itself, an eval harness by itself, a span emitter by itself. Each looks fine alone. Compose them and the seams show.

The gate chain says ALLOW but the sandbox refuses for a reason the chain did not anticipate. The eval harness records a pass but the OTel spans say the gate refused a tool the agent claims it used. The Prometheus counter is incremented twice when it should be incremented once. The observation budget is exceeded and the agent keeps going, because the chain tracked the budget and the sandbox didn't know.

This lesson is the integration test for the whole track. The agent does its work in order: read the source file, run the tests, identify the bug from the test failure, write the fix, rerun the tests, and stop. Every operation goes through the gate chain. Every tool runs against the sandbox's project root. Every step is wrapped in a span. The demo then asserts the outcome.

The agent's policy is a state machine with five working states, plus a terminal HALT. Each state maps to one tool call, and each tool call passes through the gate chain.

SURVEY: the agent reads src/fizz.py. The next state is RUN_TESTS.

RUN_TESTS: the agent runs the test command. If the tests pass, the state machine halts with success. Otherwise the next state is INSPECT.

INSPECT: the agent reads tests/test_fizz.py. The next state is FIX.

FIX: the agent writes the corrected file. The next state is VERIFY.

VERIFY: the agent runs the test command again, then the state moves to HALT whatever the result. This scaffold sets solved whenever HALT is reached, so a failing VERIFY is not reported as a failure. A real harness would check the exit code here.

If the gate chain denies a tool call, the agent records the refusal as a step, reports it in halted_reason, and halts unsolved.

The fixture bug is an off-by-one in fizz.py. The deterministic policy detects the bug with a substring check on the test failure output ("expected" present and "fizz" in the lower-cased text) and emits the corrected file. Both branches of that check emit the same hardcoded fix. Replacing the policy with an LLM does not change the harness contract.

The lesson is self-contained. Each prior-lesson primitive (gate, sandbox, ledger, span) is reimplemented at minimal scale in main.py, so the demo runs without importing its siblings. The names match lessons 25-28 exactly, which keeps the mapping back to those lessons unambiguous. Only run_tests goes through Sandbox.run. The read_file and write_file tools do their own root-path checks against sandbox.project_root.

main.py ships:

1. The minimal harness primitives, copied with the same names as lessons 25-28: GateChain, Sandbox, ObservationLedger, SpanBuilder, MetricsRegistry.
2. CodingAgentPolicy class: state machine with five working states plus a terminal HALT.
3. prepare_scratch_repo() function: prepares a scratch dir with the bundled buggy fixture.
4. AgentRun class: drives the policy, dispatches through the harness, returns an AgentRunReport.
5. A bundled fixture (fixture_repo/) with src/fizz.py and tests/test_fizz.py.
6. Demo: runs the policy end-to-end, prints the step-by-step trace, asserts pass, prints metrics.

The bundled fixture is a buggy file and a tests file, the same shape as lesson 27's tasks. The test failure message carries enough information for the deterministic policy to identify the fix. A real LLM would do the same job, slower and with broader recall, and the harness's expectations would stay the same.

A real LLM needs an API key, a network call, and tolerance for unverifiable stochasticity. The harness is the part this lesson cares about. A deterministic policy lets the demo run on any developer laptop with zero external dependencies, and lets the test suite assert exact step counts.

The policy is a strict subset of what an LLM agent does: it reads the repo, sees the failing test, identifies the line, and emits a fix. An LLM goes through the same loop under the same harness contract, and the bookkeeping is identical.

The demo exits nonzero unless four things hold, and the test suite reasserts them programmatically.

The policy solved the fixture (solved is true).

The run took fewer than 12 steps.

The observation tokens stayed within the budget.

Zero gate denials fired on legal tools (refused_legal_tool_calls == 0). The agent never invented a denied tool name.

Two more things are printed but not asserted: the number of spans emitted (one chat span plus one per tool execution, also written to traces.jsonl) and an excerpt of the Prometheus exposition, which includes tools_called_total and the tool_latency_ms histogram.

This lesson is the integration. Lesson 25 wrote the gate chain, lesson 26 the sandbox, lesson 27 the eval harness, and lesson 28 the observability. Lesson 29 shows they work as a system. A real agent harness extends from here in three swaps: a model for the deterministic policy, a real-repo task for the bundled fixture, and OTLP for the JSONL exporter.

cd phases/19-capstone-projects/29-end-to-end-coding-task-demo
python3 code/main.py
python3 -m pytest code/tests/ -v

The demo prints a run summary, a per-step trace, the span count, and a Prometheus excerpt, then exits with code zero. The tests cover the policy state transitions, the gate refusals on synthetic tool calls, the end-to-end run on the bundled fixture, and the step-budget invariants.