BuildPython

Eval-Driven Agent Development

Anthropic's guidance: "start with simple prompts, optimize them with comprehensive evaluation, and add multi-step agentic systems only when needed." Evaluation isn't the last step. It's the outer loop that drives every other choice in Phase 14.

TIME: ~60 minutesPREREQUISITES: All of Phase 14.

Learning Objectives

Name the three evaluation layers — static benchmarks, custom offline, online production — and what each is for.
Explain the evaluator-optimizer tight loop.
Describe the 2026 best practice: evals live next to code, run in CI, gate PRs.
Connect every Phase 14 lesson to the eval case it generates.

The Problem

Agents pass demos. They fail in production in ways a demo can't predict. A benchmark tells you whether a model is broadly capable, not whether this agent is shipping the right patches for your product. The fix is evaluation at three layers, running continuously, with every guardrail and learned rule mapped to an eval case.

Warm-up0/3

1. In Self-Refine (Lesson 05), what normally ends the refine loop?Warm-up

2. A team runs its automated tests on every pull request and blocks the merge if they fail. What is this practice usually called?Warm-up

3. A function that worked last week breaks after an unrelated change. Which kind of test is meant to catch this?Warm-up

Check: The Problem0/1

1. Why is a high static benchmark score not enough to ship an agent?Explain

The Concept

Three evaluation layers

Agent evaluation runs at three layers: static benchmarks, custom offline evals, and online evals in production. Each answers a different question.

Static benchmarks are off-the-shelf suites: SWE-bench Verified for code (Lesson 19), WebArena/OSWorld for browsing and desktop (Lesson 20), GAIA for generalist tasks (Lesson 19), BFCL V4 for tool use (Lesson 06). Use them for cross-model comparison and regression gating. Contamination is real: SWE-bench+ found 32.67% solution leakage. Always report Verified or audited scores.
Custom offline evals are shaped like your product:
- LLM-as-judge (Langfuse, Phoenix, Opik, Lesson 24).
- Execution-based: run the patch, check the tests.
- Trajectory-based: compare action sequences against a gold run. OSWorld-Human shows top agents taking 1.4-2.7x the gold step count.
Online evals watch production:
- Session replays (Langfuse).
- Guardrail-triggered alerts (Lesson 16, 21).
- Per-step cost and latency tracking (Lesson 23 OTel spans).

Evaluator-optimizer (Anthropic)

Evaluator-optimizer is a tight loop in which one component proposes output and another judges it, and you refine until the judge passes:

Proposer generates output.
Evaluator judges.
Refine until the evaluator passes.

This is Self-Refine (Lesson 05) generalized. Any agent flow you care about can be wrapped in it for reliability.

2026 best practice

Evals live next to the code.
They run in CI on every PR.
Merges are gated on eval scores (e.g. "no regression > 5% vs main").
Every guardrail maps to an eval case.
Every learned rule (Reflexion, pro-workflow learn-rule) maps to a failure case.

Tying Phase 14 together

Every lesson in Phase 14 generates eval cases:

Lesson	Eval case it generates
01 Agent Loop	Budget-exhausted, infinite-loop guard
02 ReWOO	Planner replans correctly when a tool fails
03 Reflexion	Learned reflections apply on retry
05 Self-Refine/CRITIC	Judge passes refined output
06 Tool Use	Argument coercion works; unknown tools rejected
07-10 Memory	Retrieval citations match sources; stale facts invalidate
12 Workflow Patterns	Each pattern produces correct output
13 LangGraph	Resume reproduces state exactly
14 AutoGen Actors	DLQ catches crashed handlers
16 OpenAI Agents SDK	Guardrail trips on the right inputs
17 Claude Agent SDK	Subagent results return to orchestrator
19-20 Benchmarks	SWE-bench Verified score, WebArena success rate, OSWorld efficiency
21 Computer Use	Per-step safety catches injected DOM
23 OTel	Spans emit required attributes
26 Failure Modes	Detectors tag known failures
27 Prompt Injection	PVE refuses poisoned retrievals
28 Orchestration	Supervisor routes to the right specialist
29 Runtime Shapes	DLQ handles N% failure

If your eval suite has a case for each row, you've covered Phase 14.

Where eval-driven development fails

No baseline. Without a last-known-good, eval scores are unreadable. Store baselines.
LLM-judge without grounding. Judges hallucinate too. Use the CRITIC pattern (Lesson 05) so the judge grounds its verdict on external tools.
Over-fitting to evals. Optimizing for the eval set drifts away from what's useful in production. Rotate cases.
Flaky evals. Non-deterministic cases cause false alarms. Pin seeds and snapshot state.

Check: The Concept0/24

1. Which set lists static benchmarks used to evaluate and compare agents?Define

2. What are the three evaluation layers the lesson names?Define

3. What is a custom offline eval?Define

4. What is an online eval?Define

5. What is the evaluator-optimizer tight loop?Define

6. What is a CI gate in eval-driven development?Define

7. What is a baseline when you track eval scores?Define

8. What does trajectory efficiency measure?Define

9. Why does the lesson treat evaluation as the outer loop rather than the last step before shipping?Explain

10. How does the evaluator-optimizer loop relate to Self-Refine?Explain

11. Which of these is a custom offline eval rather than an online eval?Explain

12. Why does the lesson say to report SWE-bench Verified or audited scores rather than raw ones?Explain

13. What is the 2026 best practice for where evals live?Explain

14. Why does the lesson warn against an LLM-judge without grounding?Explain

15. Per the lesson's Phase 14 mapping table, what eval case does Lesson 01 (Agent Loop) generate?Explain

16. What is the danger of over-fitting to evals?Explain

17. Why do flaky evals cause problems?Explain

18. Why does the lesson say evals need a stored baseline?Explain

19. On an OSWorld task the gold trajectory takes 6 steps and an agent takes 9. What is the agent's trajectory efficiency?Apply

20. Your eval pass rate rose from 70% to 92% after several prompt tweaks, but production complaints did not drop. What do you suspect first?Analyze

21. One eval case passes on a CI run and fails on the next with no code change, and the team now reruns red builds until they turn green. Which pitfall is this, and what is the fix?Analyze

22. A judge approves a refined answer that cites a paper that does not exist, because it only reads the answer text. Which change targets the cause?Analyze

23. You are shipping a coding agent. Which eval setup best fits the lesson's three layers?Design

24. You add a guardrail that blocks forwarding of social security numbers. What does the lesson's best practice say to add with it?Design

Build It

code/main.py is a stdlib eval harness with four parts:

A case registry with categories (benchmark, custom, online).
A scripted agent under test.
An evaluator-optimizer loop: propose, judge, refine until pass or max rounds.
A CI gate: aggregate pass rate plus regression against baseline.

Run it:

python3 code/main.py

The output is per-case pass/fail, a regression flag, and the CI gate verdict.

Check: Build It0/4

1. In the harness, bench_t001's proposer first returns 'patch: just craft', and its judge answers 'missing sticks in recipe'. What does evaluator_optimizer return for this case?Apply

2. ci_gate receives results where 3 of 4 cases passed, with baseline_pass_rate=0.95 and the default threshold. What does it return?Apply

3. What does ci_gate return when it is called with an empty list of results?Apply

4. A case has max_rounds=3 and its judge rejects every candidate. What does evaluator_optimizer return?Apply

Use It

Write eval cases in the same repo as your agent code.
Run them on every PR via CI.
Fail the build on regression.
Track pass rate over time.
Tie every production failure to a new case.

Check: Use It0/1

1. A production failure was just fixed with a prompt edit. What does the lesson say to do next?Explain

Ship It

outputs/skill-eval-suite.md is a skill that builds a three-layer eval suite for an agent product, with CI gates and regression tracking.

Author's Note

My bet is that the eval suite belongs ahead of the agent's structure, not after it. The motto puts evaluation before multi-step agentic systems, and I take that literally: I treat a missing eval case as a missing feature. I'd want every production failure to land in the repo as a case, and I'd want the CI gate failing the build on regression before I trusted a new agent design. I'd also keep a stored baseline, because without a last-known-good the scores can't tell me whether a change helped. And I wouldn't rely on an ungrounded judge to decide any of this, since judges hallucinate too.

Exercises

Take one of your production failures. Write an eval case that reproduces it. Does your agent pass it now?
Build an LLM-judge rubric for your domain with three dimensions (factual, tone, scope). Score 50 sessions.
Wire the eval suite into CI. Fail the build on >=5% regression.
Add a trajectory-efficiency metric: how many steps did the agent take vs a gold trajectory?
Map every Phase 14 lesson to an eval case in your suite. Anything missing is a gap to close.

Key Terms

Term	What people say	What it actually means
Static benchmark	"Off-the-shelf eval"	SWE-bench, GAIA, AgentBench, WebArena, OSWorld
Custom offline eval	"Domain eval"	LLM-as-judge / exec / trajectory on your product shape
Online eval	"Production eval"	Session replay, guardrail alerts, cost/latency tracking
Evaluator-optimizer	"Propose-judge-refine"	Iterate until judge passes
CI gate	"Merge blocker"	Fail the build on eval regression
Baseline	"Last-known-good"	Reference score to detect regression
Trajectory efficiency	"Steps over gold"	Agent step count divided by gold-trajectory step count