BuildPython

The Workbench on a Real Repo

Nine lessons of groundwork are worth nothing if they don't survive contact with a real codebase. This lesson runs the same task twice on a small sample app, prompt-only and then workbench-guided, and lets the numbers do the arguing.

TIME: ~60 minutesPREREQUISITES: Phases 14 · 32 to 14 · 40

Learning Objectives

Bring the seven workbench surfaces together on a small application.
Run the same task twice (prompt-only and workbench-guided) and measure five outcomes.
Read the before/after report and say which outcomes the workbench changed and where it is overkill.
Defend the workbench against a "but my model is good enough" pushback.

The Problem

A demo on a toy task convinces no one. The workbench earns its keep when a real-feeling task on a real-feeling repo lands in production with fewer failures, fewer reverts, and a packet the next session can use.

This lesson ships that repo and runs the same task through both pipelines. What comes out is a before/after report you can hand to a skeptic.

Warm-up0/3

1. An agent tells you "all tests pass." What is the most reliable way to confirm that?Warm-up

2. Why would a task definition list files the agent must not edit?Warm-up

3. You compare two agent setups on the same task. What must stay fixed for the comparison to be fair?Warm-up

The Concept

Mermaid diagram

A workbench-guided run is the same task executed through the seven surfaces from Lessons 34 to 40 (init script, scope contract, state, feedback runner, verification gate, reviewer, handoff). Lessons 32 and 33 set up the workbench and its instructions file that those surfaces build on. A prompt-only run skips all of them: read, edit, claim done.

The sample app

A minimal FastAPI-style handler in sample_app/:

app.py with /signup (no validation yet).
test_app.py with one happy-path test.
README.md and scripts/release.sh as forbidden-zone bait.

The task

Add input validation to /signup: reject passwords shorter than 8 characters, return 422 with a typed error envelope. Add a test that proves the new behavior.

The two pipelines

Prompt-only:

Read the README.
Read app.py.
Edit files.
Claim done.

Workbench-guided (seven surface steps plus the edit itself, eight steps in all):

Run init script (Lesson 35).
Read scope contract (Lesson 36).
Read state (Lesson 34).
Edit allowed files only.
Run acceptance command via feedback runner (Lesson 37).
Run verification gate (Lesson 38).
Run reviewer (Lesson 39).
Generate handoff (Lesson 40).

The five outcomes measured

Outcome	Why it matters
`tests_actually_run`	A "tests passed" claim means nothing unless the tests ran
`acceptance_met`	The test that proves the goal must be the test that ran (true or false)
`files_outside_scope`	Edits outside the allowed files are easy to miss in review
`handoff_quality`	The next session pays for or benefits from this
`reviewer_total`	Reviewer score out of 10, a qualitative judgment on top of the gate

Check: The Concept0/11

1. What is the sample app in this lesson's comparison?Define

2. What does "pipeline" mean in this lesson?Define

3. What is the before/after report for?Define

4. The prompt-only pipeline reads the README, reads app.py, edits files and claims done. How does the workbench-guided pipeline compare?Explain

5. Why does the sample app include README.md and scripts/release.sh when the task never needs to change them?Explain

6. Why does the report track tests_actually_run instead of trusting a pass/fail claim?Explain

7. Why does the report record acceptance_met when it already records tests_actually_run?Explain

8. What does reviewer_total add that the other four outcomes do not?Explain

9. An agent run reports "all tests pass," but tests_actually_run is False. What is the right reading?Analyze

10. A workbench run shows tests_actually_run True and acceptance_met False. What most likely happened?Analyze

11. A workbench run's diff touches sample_app/scripts/release.sh. Which surface should have stopped it, and which outcome shows the miss?Analyze

Build It

code/main.py runs both pipelines against the same sample app fixture. Both are scripted, with no LLM in the loop, so the measurement is reproducible. The outcome values for each pipeline are hard-coded in the script, so it demonstrates how the five outcomes are recorded and reported rather than measuring a live agent. The script writes its comparison to before-after-report.md and comparison.json.

Run it:

python3 code/main.py

You get a console table of outcomes per pipeline, the markdown report saved next to the script, and the JSON for anyone who wants to chart it.

Check: Build It0/6

1. Why are both pipelines in main.py scripted with no LLM in the loop?Explain

2. In run_prompt_only, the touched files are sample_app/app.py, README.md and sample_app/scripts/release.sh, and ALLOWED holds app.py and test_app.py. What does the files_outside_scope row show for prompt-only?Apply

3. What do the prompt-only column's handoff_quality and reviewer_total (/10) rows print in the report?Apply

4. Suppose run_workbench also touched sample_app/scripts/release.sh. What changes in its output?Apply

5. With the original sample handler in SAMPLE_APP_PY, what does signup("[email protected]", "short") do, and what does the task require?Apply

6. You run python3 code/main.py. Which output files does it write next to the script?Apply

Production patterns in the wild

The skeptic's question is "how much does the workbench actually help?" The 2026 numbers answer it better than any explanation I could write.

Terminal Bench Top-30 to Top-5 on the same model. LangChain's Anatomy of an Agent Harness (April 2026) describes a coding agent that jumped from outside the top 30 to rank five on Terminal Bench 2.0 by changing only the harness. Same model, different surfaces, a twenty-five-rank delta.

Vercel 80% to 100% by deleting tools. Vercel reported that deleting 80% of its agent's tools moved the success rate from 80% to 100%. A smaller tool surface means sharper scope and fewer ways to fail.

Harvey 2x accuracy via harness alone. Legal agents more than doubled their accuracy through harness optimization, with no model change.

88% of enterprise AI agent projects fail to reach production. The preprints.org Harness Engineering for Language Agents paper (March 2026) traces those failures to runtime, not reasoning: stale state, brittle retries, overgrown context, and poor recovery from intermediate mistakes.

Long-context collapse. WebAgent baseline success of 40-50% drops to under 10% in long-context conditions, mostly from infinite loops and goal loss. The handoff packet exists to absorb that.

False negatives still exist. Single-step factual tasks, one-line lints, formatter runs, and anything the model has memorized verbatim run faster prompt-only. The benchmark should list them honestly so the workbench doesn't get framed as overkill.

The takeaway isn't that the harness wins forever. Models do absorb harness tricks over time. Today, though, the engineering load sits in the seven surfaces, and these numbers show it.

Check: Production patterns in the wild0/8

1. What is a false negative in the workbench comparison, and how should the report treat it?Define

2. Why does the lesson want the report to name the tasks where prompt-only is quicker?Explain

3. A coding agent moved from outside the top 30 to rank five on Terminal Bench 2.0 with the model unchanged. What does that show?Explain

4. The 2026 preprint on harness engineering says 88% of enterprise agent projects fail to reach production. Where does it trace those failures?Explain

5. Vercel moved its agent from 80% to 100% success. What did it change, and why did that help?Explain

6. Harvey's legal agents more than doubled their accuracy. What changed?Explain

7. WebAgent baseline success falls from 40-50% to under 10% in long-context conditions. What does the lesson blame, and what addresses it?Explain

8. A teammate notes that the workbench is slower on one-line formatter runs and says to drop it entirely. Which response fits the lesson?Design

Use It

Treat this lesson as the case file. Cite it when:

Someone asks why every PR carries an agent-rules.md and a scope contract.
A team wants to drop the verification gate "just for this sprint."
A new agent product launches and you need a portable benchmark for whether it actually saves time.

The numbers travel further than the explanation.

Check: Use It0/1

1. A teammate says, "My model is good enough, let's drop the verification gate this sprint." Which reply fits this lesson?Design

Ship It

outputs/skill-workbench-benchmark.md is a portable evaluation harness. It runs any agent product through both pipelines against a project's own sample app and reports the five outcomes.

Check: Ship It0/2

1. What is a workbench benchmark?Define

2. You must judge whether a new agent product saves time on your team's code. Which design fits this lesson?Design

Author's Note

My bet is that a report earns credibility by listing its false negatives alongside its wins. Single-step factual tasks, one-line lints and formatter runs are faster prompt-only, and I'd put that in front of the skeptic before they ask. A before/after report where the workbench wins every row invites the "overkill" objection. A report that names where prompt-only is quicker, and still makes the case for the workbench, is one I'd expect a skeptic to take seriously. So I treat the overhead on those tasks as a real cost to state up front, not one to bury.

Exercises

Add a sixth outcome: time-to-first-meaningful-edit. How do you measure it cleanly?
Run the comparison on a real second-day task in your codebase. Where do the workbench numbers slip?
Add a "false negative" pass covering tasks where prompt-only would have been faster and the workbench overhead is a real cost. Defend keeping the workbench anyway.
Replace the scripted "agent" with a real LLM call. Which outcomes get noisier?
Write a one-page summary aimed at a non-engineer. What survives the cut?

Key Terms

Term	What people say	What it actually means
Sample app	"Toy repo"	Small but realistic enough to exercise all seven surfaces
Pipeline	"Workflow"	Ordered sequence of surface reads/writes the agent follows
Before/after report	"The receipts"	The artifact you hand to a skeptic
False negative	"Workbench overkill"	Tasks where prompt-only is faster; useful to enumerate honestly
Workbench benchmark	"Reliability score"	Portable harness that runs the comparison on your codebase