BuildPython

Self-Refine and CRITIC: Iterative Output Improvement

Self-Refine (Madaan et al., 2023) runs one LLM in three roles, generate, feedback, refine, and loops until the output is good. It averages +20 absolute across 7 tasks. CRITIC (Gou et al., 2023) hardens the feedback step by routing verification through external tools. In 2026 the pattern ships in every framework as "evaluator-optimizer" (Anthropic) or a guardrail loop (OpenAI Agents SDK).

TIME: ~60 minutesPREREQUISITES: Phase 14 · 01 (Agent Loop), Phase 14 · 03 (Reflexion)

Learning Objectives

State Self-Refine's three prompts (generate, feedback, refine) and explain why history matters for the refine prompt.
Explain CRITIC's critical insight: LLMs are unreliable at self-verification without external grounding.
Implement a stdlib Self-Refine loop with history and an optional external verifier.
Map this pattern to Anthropic's "evaluator-optimizer" workflow and OpenAI Agents SDK's output guardrails.

The Problem

An agent produces an answer that is almost right. A line of code has a syntax error, a summary runs too long, a plan misses an edge case. The obvious move is to have the agent critique its own output and fix it.

Self-Refine shows this works with a single model and no training data or RL. The catch is that LLMs are bad at self-verification on hard facts. CRITIC fixes that by routing the verify step through external tools: search, a code interpreter, a calculator, a test runner.

Together the two papers set the 2026 default for iterative improvement. Generate, verify (externally when you can), refine, and stop when the verifier passes.

Warm-up0/3

1. From Reflexion: what is the main weakness of an LLM scoring its own output with no external check?Warm-up

2. An agent calls the model in a loop until some condition is met. Why do most such loops also include a hard cap on iterations?Warm-up

3. You can check a function by running its unit tests or by asking its author whether it looks right. Which gives stronger evidence that it works?Warm-up

Check: The Problem0/1

1. A model drafts a summary that runs too long. Without any training, what is the simplest way to improve it?Define

The Concept

Self-Refine (Madaan et al., NeurIPS 2023)

Self-Refine is a loop where one LLM generates an output, critiques it, and rewrites it using that critique, repeating until the critique finds nothing left to fix. Three roles, one model:

generate(task)            -> output_0
feedback(task, output_0)  -> critique_0
refine(task, output_0, critique_0, history) -> output_1
feedback(task, output_1)  -> critique_1
refine(task, output_1, critique_1, history) -> output_2
...
stop when feedback says "no issues" or budget exhausted.

The detail that matters is history. The refine step sees every prior output and critique, so it doesn't repeat mistakes it already made. The paper ablates this, and quality drops sharply without it.

The headline result is +20 absolute improvement averaged across 7 tasks (math, code, acronym, dialog), GPT-4 included. No training, no external tools, one model.

CRITIC (Gou et al., arXiv:2305.11738, v4 Feb 2024)

CRITIC is Self-Refine with the feedback step replaced by verify(task, output, tools), a critique grounded in what external tools report instead of what the model believes. The reason is that self-scoring is the weak link. For factual claims it's unreliable, because a hallucination often looks convincing to the model that produced it. The tools can be:

A search engine for factual claims.
A code interpreter for code correctness.
A calculator for arithmetic.
Domain-specific verifiers: unit tests, type checkers, linters.

The verifier emits a structured critique built from tool results, and the refiner conditions on that.

CRITIC beats Self-Refine on factual tasks because the critique has something real behind it. On tasks with no external verifier (creative writing, formatting), it reduces to Self-Refine.

The stop condition

Two common shapes:

Verifier passes. An external test returns success. Use this whenever you can get it: unit tests, a type checker, a guardrail assertion.
No feedback issued. The model says the output is fine. Cheaper, but unreliable, so pair it with a max-iteration cap.

The 2026 default combines them: stop if the verifier passes, OR the model says fine AND iterations >= 2, OR iterations >= max_iterations.

Evaluator-Optimizer (Anthropic, 2024)

Anthropic's Dec 2024 post names this as one of five workflow patterns. It's Self-Refine/CRITIC in Anthropic's framing, with two roles:

Evaluator: scores the output and produces a critique.
Optimizer: revises the output given the critique.

The loop runs until the evaluator passes. The engineering detail Anthropic adds is that the evaluator and optimizer prompts should be substantially different, otherwise the model just rubber-stamps its own work.

OpenAI Agents SDK output guardrails

The OpenAI Agents SDK ships the same idea as "output guardrails." A guardrail is a validator that runs on an agent's final output. If it trips (raises OutputGuardrailTripwireTriggered), the output is rejected and the agent can retry. A guardrail can call tools (CRITIC-style) or be a pure function (Self-Refine-style).

2026 pitfalls

Rubber-stamp loops. The same model generating and critiquing in the same prompt style converges on "looks good to me." Use structurally different prompts, or a smaller, cheaper model for the critique.
Over-refinement. Every refine pass costs latency and tokens. Budget 1-3 passes, then escalate to human review.
CRITIC on trivial tasks. With no external verifier, CRITIC degenerates to Self-Refine. Don't pay latency for a stub verifier.

Check: The Concept0/27

1. What three prompts make up a Self-Refine loop?Define

2. What does CRITIC change relative to Self-Refine?Define

3. Which Anthropic workflow pattern matches Self-Refine and CRITIC in Claude-friendly language?Define

4. In a refine loop, what is the verify step?Define

5. What is refine history?Define

6. What is a rubber-stamp loop and how does the lesson recommend avoiding it?Define

7. What is the stop condition of a refine loop?Define

8. Which OpenAI Agents SDK feature validates an agent's final output and is CRITIC-shaped?Define

9. What does CRITIC route its external verification through?Define

10. Why does Self-Refine pass history to the refine step?Explain

11. What does Self-Refine's headline result show about what the method needs?Explain

12. Why does CRITIC beat Self-Refine on factual tasks?Explain

13. What happens to CRITIC on tasks without an external verifier, like creative writing or formatting?Explain

14. In the Self-Refine pseudocode, what does the refine step receive so it can revise rather than start over?Explain

15. What stop condition does the lesson recommend for 2026 evaluator-optimizer loops?Explain

16. Why does Anthropic say the evaluator and optimizer prompts should be substantially different?Explain

17. Why does the lesson suggest a smaller, cheaper model as the critic?Explain

18. When an OpenAI Agents SDK output guardrail trips, what happens to the agent's output?Explain

19. Why does the lesson cap refine loops at roughly 1-3 passes?Explain

20. A loop stops when the verifier passes, OR the model says fine AND iterations >= 2, OR iterations >= max_iterations. After pass 1 the verifier fails but the model says "no issues" (max_iterations is 4). What happens?Apply

21. Your loop's critic is the same model with the same prompt, and it approves a summary containing an invented statistic. What is the best fix?Analyze

22. Every task in your pipeline ends after pass 1 with "looks good", yet reviewers keep finding errors. Generator and critic are the same model with the same prompt style. What is the likely cause and fix?Analyze

23. A team ships a CRITIC loop for creative writing, with a stub verifier that only asks the model whether the text looks fine. Latency rose and quality did not. Why?Analyze

24. A loop runs 8 refine passes per request. Cost is high and later passes barely change the output. What should the team do?Analyze

25. You build an agent that writes Python functions, and the project has a test suite. Which loop design fits the lesson best?Design

26. You must add a quality loop to a tagline generator. No test, lookup, or schema check exists, and latency matters. What does the lesson's guidance imply?Design

27. Using the OpenAI Agents SDK, you want answers containing arithmetic checked by a calculator tool, with a retry when they are wrong. Which design maps best?Design

Build It

code/main.py implements Self-Refine and CRITIC on a toy task: produce a short bullet list for a topic. The verifier checks format (3 bullets, each under 60 chars). CRITIC adds an external "fact verifier" that penalizes known hallucinations.

Components:

generate: scripted producer.
feedback: LLM-style self-critique.
verify_external: CRITIC-style grounded verifier.
refine: rewrites output given history.
Stop condition: verifier passes or max 4 iterations.

Run it:

python3 code/main.py

Compare the two runs. CRITIC catches a factual error that Self-Refine misses, because the external verifier has grounding the self-critic lacks.

Check: Build It0/5

1. In code/main.py, the CRITIC run starts from a draft that says Paris is the capital of Germany and Mt Everest is in Europe. How does it end?Apply

2. In code/main.py, the Self-Refine run uses feedback_self. How does it end?Apply

3. In code/main.py, verify_external gets this two-line output: "- Paris is the capital of France" and "- Water boils at 100C". What does it return?Apply

4. In code/main.py, run_loop is called with use_critic=True and max_iters=1 on the flawed first draft. How does it end?Apply

5. In the toy Self-Refine run, the self-critic flags the first bullet on every pass, yet the output never changes. What explains this?Analyze

Use It

Anthropic's evaluator-optimizer is this pattern in Claude-friendly language. OpenAI Agents SDK output guardrails are CRITIC-shaped, since guardrails can call tools. LangGraph ships a reflection node that reads like Self-Refine. Google's Gemini 2.5 Computer Use adds a per-step safety evaluator, a CRITIC variant that verifies every action before it commits.

Check: Use It0/1

1. What makes Gemini 2.5 Computer Use's per-step safety evaluator a CRITIC variant?Explain

Ship It

outputs/skill-refine-loop.md configures an evaluator-optimizer loop from task shape, verifier availability, and iteration budget. It emits prompts for the generator, evaluator/verifier, and optimizer, plus a stop policy.

Check: Ship It0/1

1. The refine-loop skill configures an evaluator-optimizer loop from which inputs?Define

Author's Note

The first thing to check in any refine loop is whether the critic can actually be wrong in a way the generator can't. If both are the same model reading the same context, "no issues found" tells you almost nothing. A loop can burn three passes polishing prose around a fact that was invented in pass zero. Put a real check in the loop (a test, a lookup, a schema validator) even if it's crude, and the loop starts earning its latency. Without one, cap it at one pass and move on.

Exercises

Run the toy with max_iterations=1. Does CRITIC still help?
Replace the external verifier with a noisy one (random 30% false positives). What does the loop do? This is the 2026 reality of most guardrail stacks.
Implement a generator-critic split across models: big model generates, small model critiques. Does it beat same-model?
Read CRITIC Section 3 (arXiv:2305.11738 v4). Name the three verification-tool categories and give an example for each.
Map OpenAI Agents SDK's output_guardrails to CRITIC's verifier role. What does the SDK get wrong, and what does it get right?

Key Terms

Term	What people say	What it actually means
Self-Refine	"LLM that fixes itself"	Generate -> feedback -> refine loop in one model, with history
CRITIC	"Tool-grounded verification"	Replace feedback with an external verifier (search, code, calc, tests)
Evaluator-Optimizer	"Anthropic workflow pattern"	Two roles, evaluator scores and optimizer revises, looped to convergence
Output guardrail	"Post-hoc check"	OpenAI Agents SDK validator that runs after an agent produces output
Verify step	"Critique phase"	The load-bearing decision: grounded or self-rated
Refine history	"What the model already tried"	Prior outputs + critiques prepended to the refine prompt; drop it and quality collapses
Rubber-stamp loop	"Self-agreement failure"	Same-prompt critique returns "looks good"; fix with structurally different prompts
Stop condition	"Convergence test"	Verifier passes OR no feedback AND iteration cap; never a single condition