BuildPython

Runtime Feedback Loops

An agent that does not see real command output is guessing. A feedback runner captures stdout, stderr, exit code, and timing into a structured record the next turn can read, so the agent reacts to facts instead of its own prediction of them.

TIME: ~50 minutesPREREQUISITES: Phase 14 · 32 (Minimal Workbench), Phase 14 · 35 (Init Script)

Learning Objectives

Distinguish runtime feedback from observability telemetry.
Build a feedback runner that wraps shell commands and persists structured records.
Truncate large outputs deterministically so the loop stays within token budget.
Refuse to advance the loop when feedback is missing.

The Problem

The agent says "running tests now." The next message says "all tests pass." No test ran. Either the agent imagined the output, or it ran the command and never read the result, or it read the result and silently dropped the failure line.

A feedback runner closes that gap. Every command goes through it, and every record carries the command, the captured stdout and stderr, the exit code, the wall-clock duration, and a one-line agent note. The agent reads the record on the next turn. The verification gate reads all of them at the end of the task.

Warm-up0/3

1. An agent says "all tests pass" and you suspect it never ran them. Which evidence settles it?Warm-up

2. A command prints 50 MB of logs straight into an agent's context. What is the main problem?Warm-up

3. A program finishes and hands a status number back to the shell. What does a status of 0 conventionally mean?Warm-up

Check: The Problem0/1

1. What gap does routing every command through a feedback runner close?Explain

The Concept

Mermaid diagram

A feedback record is the structured entry the runner writes for each command: the command itself, the captured stdout and stderr, the exit code, the duration, and a one-line agent note. The agent reads it on the next turn instead of trusting its memory of what the command should have printed.

What goes in a feedback record

Field	Why it matters
`command`	Exact argv, no shell expansion surprises
`stdout_tail`	First 5 and last 30 lines, middle dropped behind a marker
`stderr_tail`	Same head-and-tail rule, kept separate from stdout
`exit_code`	The unambiguous success signal
`duration_ms`	Surfaces slow probes and runaway processes
`started_at`	Timestamp for replay
`agent_note`	One line the agent writes about what it expected

Truncation is deterministic

A 50 MB log wrecks the loop. The runner keeps the first 5 and last 30 lines and drops the middle behind a ...truncated N lines... marker. (The field names say tail because the tail carries most of the signal, but each holds both ends.) The rule is deterministic, so the same output always produces the same record. There is no sampling, and the lines the agent needs (the final error, the final summary) sit at the tail.

Feedback versus telemetry

Telemetry (Phase 14 · 23, OTel GenAI conventions) serves human operators reviewing runs across time. Feedback serves the next turn of this run. They share fields but live in different files with different retention.

Refuse to advance without feedback

If the runner errors before it captures an exit, the record carries exit_code: null and error: <reason>. The agent loop must not claim success on a null exit. No exit, no progress.

Check: The Concept0/14

1. What must the loop do when a record has `exit_code: null`?Define

2. How does the runner truncate large outputs?Define

3. How does feedback differ from telemetry?Define

4. What is a feedback record?Define

5. What is the agent note in a feedback record?Define

6. Why do feedback and telemetry live in different files instead of one shared log?Explain

7. Why is exit_code the field that decides whether the loop advances?Explain

8. Why is the truncation rule deterministic rather than sampled?Explain

9. Why does the agent write its note before it reads the result?Explain

10. A test run exits 1 after 800 lines, and the failing assertion is printed last. Why does the head-and-tail rule still serve the agent?Explain

11. The verification gate finds that the agent reported the test step as passed, but the record shows `exit_code: null` and `error: timeout after 30.0s`. What went wrong?Analyze

12. A test run printed 800 lines and its first failing test appeared at line 300. The agent cannot find it in the record. What explains this?Analyze

13. A record has agent_note "expect all tests pass" and exit_code 1, yet the agent's next message says the tests passed. What does this show?Analyze

14. A team wants one shared file to feed the agent's next turn and the operators' cross-run dashboards. Which response fits the lesson?Design

Build It

code/main.py implements three pieces:

run_with_feedback(command, agent_note) wraps subprocess.run, captures stdout, stderr, exit code, and duration, truncates deterministically, and appends to feedback_record.jsonl.
A small loader streams the JSONL into a Python list.
A demo runs five commands (success, secret leak, failure, a retry of the failure, missing binary), prints a summary line for each, and reconstructs the retry chain.

Run it:

python3 code/main.py

The output is five feedback records appended to feedback_record.jsonl. Tail the file across re-runs to watch the loop accumulate.

Check: Build It0/6

1. `deterministic_tail` runs with its defaults on an output of 100 lines. What does the result contain?Apply

2. A command prints exactly 35 lines. What does `stdout_tail` hold under the default budget?Apply

3. A command prints the line `password=hunter2`. What does the stored `stdout_tail` show for it?Apply

4. `run_with_feedback` is called with a binary that does not exist. What does the record hold, and what does `loop_can_advance` return?Apply

5. A command hangs and the runner's timeout fires before the process exits. What should the record and the loop do?Apply

6. `fail` has id A. `retry` has id B and `parent_command_id` A. What does `retry_chain(B)` return?Apply

Production patterns in the wild

Three patterns harden the runner enough to ship.

Redact at write, not at read. Any record that touches stdout or stderr can leak secrets, so the runner applies a redaction pass before the JSONL append. It replaces the secret itself, not the whole line, for matches of Bearer <token>, AKIA[0-9A-Z]{16} (AWS), xox[baprs]- (Slack), and PEM private-key blocks (the whole block becomes [REDACTED PRIVATE KEY]). One more pattern covers password, passwd, secret, api_key, access_key and token followed by = or : and a value. The key name stays, the separator is normalised to = (so password: hunter2 is stored as password=[REDACTED]), and the value becomes [REDACTED]. Redacting at read time is a foot-gun because the file on disk is what an attacker reaches. Audit the patterns quarterly against the secret formats your production runtime actually produces.

Rotate, don't grow a single file. Cap feedback_record.jsonl at 1 MB per file. On overflow, rotate to .1, .2, and drop .5. The agent's loop reads only the current file, so the runtime cost stays bounded, while CI artifact storage keeps the full rotated set. Without rotation, the file becomes the bottleneck on every loader call.

Chain retries with a parent id. Every record gets a command_id, and a retry carries parent_command_id pointing at the previous attempt. The reviewer's "failed attempts" list (Phase 14 · 39) and the verification gate's audit both follow that chain. Without the link, retries look like independent successes and the audit hides the failure history.

Check: Production patterns in the wild0/5

1. Why redact secrets at write time rather than read time?Explain

2. Why cap `feedback_record.jsonl` at 1 MB with rotation?Explain

3. An agent retries a failing build three times and the third attempt passes. Which field lets an audit see the earlier failures?Apply

4. The loader shows `[REDACTED]` for a key, yet someone who opens the JSONL on disk finds the live key. What went wrong?Analyze

5. You add a migration step that sometimes hangs and prints thousands of lines containing `password=` values. The agent must act on the results. Which setup fits?Design

Use It

Where this shows up in practice:

Claude Code Bash tool. The tool already captures stdout, stderr, exit, and duration. The runner in this lesson is the framework-agnostic equivalent for any agent product.
LangGraph nodes. Wrap any shell node in the runner so the record persists outside graph state.
CI logs. Pipe the JSONL into your CI artifact store, and reviewers can replay any command without rerunning the session.

The runner is a thin wrapper, and it survives framework migrations because it owns the shape of the record.

Ship It

outputs/skill-feedback-runner.md generates a project-specific run_with_feedback.py with the right truncation budget, a JSONL writer wired to the workbench, and a loader the agent reads at every turn.

Author's Note

I treat exit_code as the field that decides whether the loop advances. If the runner can't capture it, the record says null and I stop the loop there, because an agent that can't show how a command ended has nothing to base "it passed" on. I'd take a halted run with a clear error over a confident summary of output nobody captured. The runner exists to close the gap between what the agent says happened and what was recorded, and a null exit is that gap showing up in the file.

Exercises

Add a cwd field to each record so the same command run from different directories is distinguishable.
Add a redaction step that replaces the secret in Bearer <token> and password=<value> matches with [REDACTED]. Test it on a fixture record.
Cap total feedback_record.jsonl size at 1 MB by rotating to .1, .2 files. Defend the rotation policy.
Add a parent_command_id so retry chains are visible: which command produced the input that the next command consumed.
Pipe the JSONL into a tiny TUI that highlights the latest non-zero exit. List the eight key features the TUI must show to be useful in a review.

Key Terms

Term	What people say	What it actually means
Feedback record	"Run log"	Structured JSONL entry with command, output, exit, duration
Tail truncation	"Trim the log"	Deterministic capture of the first 5 and last 30 lines so records fit in token budget
Refuse-on-null	"Block on missing data"	The loop must not advance when `exit_code` is null
Agent note	"Expectation tag"	The one-line prediction the agent writes before reading the result
Telemetry split	"Two log files"	Feedback for the next turn, telemetry for the operator