Reflexion: Verbal Reinforcement Learning

When an agent fails a task, standard RL says to run thousands more trials, compute gradients, and update weights. That is slow and expensive, and most production agents have no training budget for every failure.

Reflexion (Shinn et al., arXiv:2303.11366) takes a cheaper route. The agent reads its own failed attempt, writes down why it failed, and retries with that note in its prompt. No weight updates, no gradients, just natural language carried between trials.

It works. On ALFWorld it beats ReAct and the other non-fine-tuned baselines. On HotpotQA it improves over ReAct. On code generation (HumanEval/MBPP) it set state of the art at the time. Not one gradient step involved.

Reflexion is a pattern for learning from failure without touching model weights: after a failed trial, the agent writes a natural-language reflection on what went wrong, stores it, and reads it at the start of the next trial. The paper calls this verbal reinforcement learning, because the reflection plays the role the gradient plays in ordinary RL.

The three components

Actor         : generates a trajectory (ReAct-style loop)
Evaluator     : scores the trajectory — binary, heuristic, or self-eval
Self-Reflector: writes a natural-language reflection on the failure

Plus one data structure:

Episodic memory: list of prior reflections, prepended to the next trial's prompt

A trial runs the Actor, and the Evaluator scores the result. On a low score the Self-Reflector produces a reflection ("I picked the wrong tool because I read the question as asking about X when it asked about Y"). That reflection goes into episodic memory. The next trial starts fresh, with no leftover trajectory, but it sees the reflection.

Three evaluator types

1. Scalar. An external binary signal: ALFWorld succeeds or fails, HumanEval tests pass or fail. The simplest option and the highest-signal one.
2. Heuristic. Predefined failure signatures. "Same action twice in a row means stuck." "More than 50 steps means inefficient."
3. Self-evaluated. The LLM scores its own trajectory. You need this when no ground truth exists, but the signal is weaker, so pair it with tool-grounded verification (Lesson 05, CRITIC).

The 2026 default is a mix: scalar when you have it, self-eval when you don't, heuristics as safety rails.

Why this generalizes

Reflexion is less a new algorithm than a named pattern, and almost every production "self-healing" agent runs some variant of it:

• Letta's sleep-time compute (Lesson 08): a separate agent reflects on past conversations and writes to memory blocks.
• Claude Code's CLAUDE.md and "save memory" flow: learnings are captured and prepended to future sessions.
• pro-workflow's /learn-rule command: corrections become explicit rules.
• LangGraph's reflection nodes: a node scores the output and routes to a refine step when needed.

They all rest on one insight. Natural language is rich enough to carry "what I learned from failure" from one run to the next.

When it works and when it does not

Reflexion works when:

• There is a clear failure signal (a failing test, a tool error, a wrong answer).
• The task class is reproducible, so the same type of question can come up again.
• The agent has enough action budget to actually act on the reflection.

It does not help when:

• The agent already succeeds on the first try.
• The failure is external. A network outage or a broken tool teaches nothing about the next run.
• The reflection hardens into superstition, a story built around one flaky run.

The 2026 pitfall is memory rot. Reflections pile up, some go stale or were wrong to begin with, and every re-run gets slower as the episodic buffer grows. The fixes are periodic compaction (Lesson 06), a TTL on reflections, or a separate sleep-time cleanup agent (Letta).

code/main.py implements Reflexion on a toy puzzle: produce a 3-element list that sums to a target. The Actor emits candidate lists, the Evaluator checks the sum, and the Self-Reflector writes a line about what went wrong. That line goes into episodic memory for the next trial.

Components:

• Actor: a scripted policy that improves when it sees reflections.
• Evaluator.binary(): pass/fail on the target sum.
• SelfReflector: generates a one-line diagnosis of the failure.
• EpisodicMemory: a bounded list with TTL semantics.

Run it:

python3 code/main.py

The trace shows three trials. Trial 1 fails and a reflection is stored. Trial 2 sees the reflection and improves but still fails. Trial 3 succeeds. Compare that with the baseline run, which has no reflection and stays stuck on trial 1's answer.

LangGraph ships reflection as a node pattern. Claude Code's /memory command and pro-workflow's /learn-rule externalize the episodic buffer as a markdown file. Letta's sleep-time compute runs the Self-Reflector during downtime, so the primary agent keeps its latency budget. The OpenAI Agents SDK has no built-in Reflexion. You assemble it from a custom Guardrail that rejects trajectories by score and a memory Session that survives across runs.

outputs/skill-reflexion-buffer.md creates and maintains an episodic buffer with reflection capture, TTL, and deduplication. Given a task class and a failure, it emits a reflection that helps the next trial, not a generic "be more careful."

1. Switch from the binary evaluator to a scalar one that returns a distance metric (how far the sum is from the target). Does it converge faster?
2. Add a TTL of 10 trials to reflections. Do older reflections hurt or help after that point?
3. Implement a heuristic evaluator that marks a trial as stuck when the same action repeats. How does it interact with the Self-Reflector?
4. Run Reflexion with an adversarial Actor that ignores reflections. What is the least reflection-prompt engineering that forces the Actor to notice them?
5. Read Section 4 of the Reflexion paper on ALFWorld. Reproduce the 130% success-rate improvement conceptually: what is the key delta versus vanilla ReAct?

• Shinn et al., Reflexion: Language Agents with Verbal Reinforcement Learning (arXiv:2303.11366): the canonical paper
• Letta, Sleep-time Compute: async reflection in production
• Anthropic, Effective context engineering for AI agents: managing the episodic buffer as part of context
• LangGraph overview: reflection node pattern