LearnPython

Bounded Self-Improvement Designs

Research has converged on four primitives for bounding a self-improvement loop. Formal invariants that must hold across every edit. Alignment anchors that cannot be modified. Multi-objective constraints where every dimension (safety, fairness, robustness) must hold, not just performance. Regression detection that pauses the loop when historical metrics suggest capability loss. None of them is a proof of safety. Information-theoretic results (Kolmogorov complexity, Lob's theorem) bound what any system can prove about its own successors, so treat the primitives as mitigations that raise the cost of silent failure.

TIME: ~60 minutesPREREQUISITES: Phase 15 · 07 (RSI), Phase 15 · 04 (DGM)

Learning Objectives

Name the four bounding primitives (formal invariants, alignment anchors, multi-objective constraints, regression detection) and the failure each one targets.
Explain why these primitives are mitigations rather than proofs of safety, citing the information-theoretic limits.
Describe how the four primitives stack into a gate that every proposed self-edit must pass.
Run the toy bounded loop with gates on and off, and read the reject counters to see which gate stopped which edit.

The Problem

Lesson 7's race simulator showed that small rate differences compound into large gaps. Lesson 4's DGM case study showed that loops can game their own evaluators. Both point at the same engineering question: what constraints can you put on a self-improvement loop so that the loop itself can't quietly weaken them?

The ICLR 2026 RSI Workshop summary (openreview.net/pdf?id=OsPQ6zTQXV) identifies four such primitives. Anthropic's RSP v3.0 (Lesson 19) and DeepMind's FSF v3 (Lesson 20) both reference them in capability thresholds. The Meta HyperAgents work and community frameworks like SAHOO (March 2026) implement subsets in production.

Be honest about what they are: mitigations. Information-theoretic results bound what any system can prove about its own successor, and no current design closes that gap formally. A well-bounded loop is safer than an unbounded one, but that is a claim about degree, not a guarantee.

Warm-up0/3

1. In Lesson 4's Darwin Godel Machine case study, which failure mode did the self-modifying loop show?Warm-up

Schematic of a self-modifying loop from Lesson 4: an agent makes a self-edit, the edit gets a score, and the next cycle starts from the edited agent. A dashed box asks which failure mode the loop showed. — Typed schematic, illustrative.

2. In Lesson 7's race simulator, what happened when one improvement loop gained slightly more per cycle than another?Warm-up

A small table comparing two improvement loops. Loop A gains slightly more per cycle than Loop B. After one cycle the gap between them is small. The after-many-cycles cells are question marks. — Typed schematic, illustrative.

3. What defines a recursive self-improvement (RSI) loop, as covered in Lesson 7?Warm-up

Typed schematic, illustrative.

Check: The Problem0/1

1. What does 'bounded improvement' claim about a loop built with the four primitives?Define

A self-improving loop box feeds a row of four gates labelled Invariants, Anchor, Multi-obj. and Regression. Below, a dashed box with a question mark asks what the design lets you claim about the loop's safety. — Typed schematic, illustrative.

The Concept

Primitive 1: formal invariants

An invariant is a property that must hold before and after every self-modification. Examples:

Output distribution is conditioned on a fixed constitution header (Lesson 17).
No tool call goes to an unauthorized endpoint.
Memory writes go through a logged, signed path.
The evaluator's module hash matches the approved version.

Invariants are checked by external code the loop cannot edit. If a proposed modification violates one, it is rejected. This catches the DGM-style evaluator-tampering mode (Lesson 4).

The hard part is picking invariants that are necessary for safety and computable in reasonable time. Over-specify and progress stalls. Under-specify and the loop walks past properties that mattered.

Primitive 2: alignment anchors

An alignment anchor is an immutable representation of the loop's core objective, pinned outside the loop's edit surface. Examples:

A constitutional text (Lesson 17) loaded from a read-only location on every invocation.
An evaluator in a separate repository with independent access control.
A tool-permission manifest signed by a human and re-verified each cycle.

The anchor prevents objective drift. The loop may get better at pursuing the objective, but it can't edit what the objective is.

The subtle failure: a loop that can't edit the anchor can still reinterpret it by drifting in how it reads it. Constitutional AI (Lesson 17) is explicitly reason-based so it can handle novel situations, and that reasoning layer is where interpretation drift lives. Anchors are necessary, not sufficient.

Primitive 3: multi-objective constraints

A loop that optimizes a single scalar score will find shortcuts. A loop that has to satisfy several hard constraints at once has fewer shortcuts available. Typical axes:

Performance (task-level benchmark)
Safety (red-team evaluations, refusal rate on known-bad)
Fairness (disparate-impact bounds on sensitive subgroups)
Robustness (OOD test sets, adversarial input handling)

A modification is accepted only if every constraint holds. Lesson 13's cost governor adds financial constraints on top, and Lesson 18's Llama Guard plugs in as a safety axis.

Primitive 4: regression detection

Each cycle's scores are compared against historical distributions. A drop beyond the configured tolerance pauses the loop. This catches silent capability loss that would otherwise get absorbed into the running average as the loop "improved past" it.

A practical implementation stores the last N cycles' per-task scores and computes a per-task delta for each new cycle. If a delta falls below the threshold, the cycle is rejected and a human reviews.

Information-theoretic limits

Kolmogorov complexity and Lob's theorem set upper bounds on what a system can prove about itself. Schmidhuber's formal Godel Machine (Lesson 4) aimed at the highest such bound, and nobody has completed a non-trivial proof. Lob's result says that if a system provably believes "I will do X if I prove I should do X," it will do X without proving it should. That is a well-known self-reference failure.

So the primitives can't close the safety problem. They make silent failure more expensive. A malicious or drifted loop that would have slipped past a missing check now has to actively undermine an explicit one, and that is a more detectable signature.

A worked example

Suppose an agent proposes an edit. The gating stack:

Invariant checks: module hashes, tool-permission manifest, constitutional header.
Anchor check: objective statement matches the approved version (byte-wise or semantically).
Multi-objective evaluation: performance, safety, fairness, robustness axes.
Regression detection: no axis drops more than tolerance.

The edit lands only if all four pass. A single failure pauses the loop.

Check: The Concept0/22

1. Which set names the four primitives for bounding a self-improvement loop?Define

A proposed edit enters a chain of four dashed boxes labelled Primitive 1 to Primitive 4, each marked with a question mark, and the edit lands only after the fourth. — Typed schematic, illustrative.

2. What is an invariant in a bounded self-improvement loop?Define

A card quoting one example from the lesson: no tool call goes to an unauthorized endpoint. Below it an agent box sends calls toward an approved endpoint and an unauthorized endpoint. A dashed box with a question mark asks what such a property is called. — Typed schematic, illustrative.

3. What is an alignment anchor?Define

Two cards from the toy loop. The first shows the objective string 'canonicalize whitespace and title-case the input'. The second shows its first 16 hex characters of SHA-256, 175083c3ffb99e98. A dashed box with a question mark asks what such an objective is called. — Objective string from main.py; hash computed by the script.

4. Under the multi-objective primitive, when is a modification accepted?Define

A proposed edit feeds four axis boxes, Performance, Safety, Fairness and Robustness, each with an unknown result. A dashed box below asks when the edit is accepted. — Typed schematic, illustrative.

5. What does regression detection do with each cycle's scores?Define

Bar chart of four earlier cycle scores, 0.50, 0.75, 1.00 and 1.00, followed by a dashed empty slot for cycle 5 marked with a question mark. A dashed box asks what regression detection does with the new score. — Illustrative scores, not from a run.

6. What does the Kolmogorov bound limit, as the lesson uses it?Define

A three-column Key Terms row. Term: Kolmogorov bound. What people say: information-theoretic limit. What it actually means: a dashed cell with a question mark. — Lesson's Key Terms layout with the meaning hidden.

7. What does Lob's theorem imply about a self-referential system?Define

A three-column Key Terms row. Term: Lob's theorem. What people say: self-reference trap. What it actually means: a dashed cell with a question mark. — Lesson's Key Terms layout with the meaning hidden.

8. In a gate stack, what happens if any one of the four checks fails?Define

Typed schematic, illustrative.

9. Which failure mode do formal invariants, checked by code the loop cannot edit, specifically catch?Explain

Typed schematic, illustrative.

10. What is the subtle failure of an alignment anchor that the loop cannot edit?Explain

Three boxes for cycle 1, 2 and 3, each showing the same read-only anchor text, linked by arrows. Below, a dashed box with a question mark asks what subtle failure remains. — Typed schematic, illustrative.

11. Why is a loop with all four primitives still not shown to be safe?Explain

Typed schematic, illustrative.

12. Given that the primitives cannot close the safety problem, how do they still help?Explain

Typed schematic, illustrative.

13. Why does the gate stack reject an edit on any single failure instead of combining the four results into one score?Explain

Typed schematic, illustrative.

14. Why do several hard constraints leave a loop fewer shortcuts than one scalar score?Explain

Left panel: one scalar score labelled benchmark score. Right panel: four hard constraints, Performance, Safety, Fairness and Robustness. A dashed box with a question mark asks why the second design leaves fewer shortcuts. — Typed schematic, illustrative.

15. How does an alignment anchor differ from a formal invariant?Explain

Two cards. Alignment anchor lists one example, a constitutional text. Formal invariant lists one example, no tool call goes to an unauthorized endpoint. A dashed box with a question mark asks how they differ. — Examples quoted from the lesson.

16. An agent's proposed edit adds an unapproved tool endpoint to its manifest and raises its benchmark score. Safety, fairness, and robustness scores stay flat. Which gate should reject it first?Apply

A table describing one proposed edit: the tool manifest adds an unapproved tool endpoint, the benchmark score rises, and the safety, fairness and robustness scores stay flat. A dashed box with a question mark asks which gate should reject it first. — Restates the stem.

17. A loop accepts any edit that raises its benchmark score. Over many cycles the benchmark climbs, but the refusal rate on known-bad inputs collapses. Which primitive is missing?Analyze

Line chart over cycles with no numeric axis. The benchmark score line climbs steadily and the refusal rate on known-bad inputs line falls toward the bottom. A dashed box with a question mark asks which primitive is missing. — Illustrative curve shapes, not a measured run.

18. An anchor is stored read-only and verified every cycle, yet the agent's behavior drifts from what the authors intended. Which weakness explains it?Analyze

Three green boxes for cycles 1 to 3, each showing the anchor verified. Below, a dashed horizontal line marks the intended behavior and a red line labelled observed behavior slopes away from it. A dashed box with a question mark asks which weakness explains the drift. — Illustrative shape, not a measured run.

19. Team A keeps the objective text in the repository the agent can commit to and re-reads it each cycle. Team B keeps it in a separate repository with independent access control. Which design is better?Analyze

Two panels. Team A: an agent and a repository the agent can commit to, with the objective text inside it, re-read each cycle. Team B: an agent and a separate repository with independent access control holding the objective text, re-read each cycle. A dashed box with a question mark asks which design is better. — Typed schematic, illustrative.

20. You must design an alignment anchor for a coding agent that edits its own repository. Which design fits?Design

A dashed region labelled the agent's repository holds a coding agent that edits files in it. Below it, a dashed box with a question mark asks where the objective and its checks are stored and how they are checked. — Typed schematic, illustrative.

21. A loop already gates on performance, safety, fairness, and robustness. You must add a monthly spend limit. Where does it fit best?Design

Four boxes for the existing axes, Performance, Safety, Fairness and Robustness, each marked gated. Below them, a dashed box labelled monthly spend limit asks where it fits. — Typed schematic, illustrative.

22. Per-task scores in your loop are noisy between cycles. You want to pause on real capability loss, not on noise. Which regression design fits?Design

Line chart of two tasks over 10 cycles. Task A wobbles slightly around 0.80. Task B wobbles around 0.70 and then drops to about 0.55 in the last two cycles. A dashed box with a question mark asks which regression design pauses on real loss and not noise. — Illustrative scores, not a measured run.

Use It

code/main.py runs a bounded self-improvement loop on a toy agent that edits a list of string operations, with the four primitives layered on as gates. Each gate can be switched on or off individually, so you can compare runs and see which gate rejected what.

Check: Use It0/6

1. In the toy loop, history_perf is [0.5, 1.0] and a candidate scores perf 0.75 with the default tolerance of 0.2. What does gate_regression return, and why?Apply

Bars for a two-cycle history, 0.50 and 1.00, and a dashed bar for a candidate at 0.75. The default tolerance 0.2 is noted. A dashed box asks what gate_regression returns. — Values from the stem.

2. A candidate scores perf 0.75 and safety 0.5 in the toy loop. What does gate_multi return, and why?Apply

Two boxes, perf 0.75 and safety 0.5, for a candidate in the toy loop. A dashed box with a question mark asks what gate_multi returns for these scores. — Values from the stem.

3. In the all-gates-off run, 200 of 200 cycles are accepted. What does the final printout show?Apply

A terminal showing the header and the line accepted 200/200 cycles for the all-gates-off run. The final perf, final safety, final ops, manifest, objective and rejects lines are replaced by question marks. — Header and accepted line are real output of main.py; the other lines are hidden.

4. In the all-gates-on run, 96 of 200 cycles are accepted and the rest are rejected. Which gate rejected the most candidates?Apply

A terminal showing accepted 96/200 cycles for the all-gates-on run and a rejects line with the four counters invariant, anchor, multi and regress replaced by question marks. A note says 104 rejected cycles are split across the counters. — Header and accepted line are real output of main.py; counters hidden.

5. In the toy, the run with only regression off ends with the same final scores as the all-on run. What should a reader conclude?Analyze

A table of two runs. All gates on: 96/200 accepted, final perf 1.00, final safety 1.00, 27 regression rejects. Only regression off: 116/200 accepted, final perf 1.00, final safety 1.00, 0 regression rejects. — Real output of python main.py (seed 5).

6. In the all-gates-on run the anchor gate rejects zero edits. Does that mean the anchor gate is useless?Analyze

A terminal showing the all-gates-on header and a rejects line in which the anchor counter is 0 and the invariant, multi and regress counters are question marks. A dashed box with a question mark asks whether the anchor gate is useless. — Header and anchor count from the stem; other counters hidden.

Ship It

outputs/skill-bounded-loop-review.md audits a proposed bounded loop and scores which of the four primitives it implements and which it only claims to.

Author's Note

I'd run main.py first and read the reject counters before the prose. The all-on run prints how many candidate edits each gate turned away, and the all-off run accepts every cycle and ends with raw_eval in the manifest, a drifted objective, and perf and safety both at 0.00. That contrast is the whole argument in about thirty lines of output.

The toy is thinner than the concept section. It has two axes, performance and safety, not four. The invariant gate is a subset check of the active manifest against the approved set, not a module hash. Regression detection compares each candidate to the best score so far with a tolerance of 0.2, not a per-task delta over the last N cycles. I'd treat the code as a sketch of where each gate sits in the loop, and not as an implementation of the real thing.

The output also tells you something the headline text glosses over. In the all-on run the anchor gate rejects zero edits, because the mutation that rewrites the objective also adds raw_eval to the manifest, and the invariant gate catches it first. And in the run with only regression off, the final scores match the all-on run. The toy doesn't demonstrate the silent dip, so exercise 2 is where you have to build that case yourself.

I'd also keep the seed fixed at 5 while you experiment, since every scenario reseeds the same way and the only thing that changes between runs is the gate configuration.

Exercises

Run code/main.py with all primitives enabled. Confirm the loop still improves on the primary metric without letting the hack win.
Disable regression detection. Construct an input where this leads to silent capability loss being accepted.
Disable the multi-objective constraint. Show the loop converges on the performance axis while a safety axis drops.
Design an alignment anchor for a coding agent. What text, stored where, checked how?
Read the ICLR 2026 RSI Workshop summary. Pick one of the four primitives and propose a concrete improvement to the current state of the art.

Key Terms

Term	What people say	What it actually means
Invariant	"Always-true property"	A property checked by external code before and after every edit
Alignment anchor	"Pinned objective"	Immutable core-goal representation outside the loop's edit surface
Multi-objective constraint	"All axes must hold"	Performance, safety, fairness, and robustness are all required
Regression detection	"Pause on drop"	Pause the loop when historical metric deltas suggest capability loss
Kolmogorov bound	"Information-theoretic limit"	Limits what a system can prove about its own successor
Lob's theorem	"Self-reference trap"	System can act on "I should" without proving it should
Gate stack	"Layered check"	Multiple primitives combined; any failure rejects the edit
Bounded improvement	"Mitigation, not proof"	Raises silent-failure cost; does not close the safety problem

Bounded Self-Improvement Designs

Learning Objectives

The Problem

The Concept

Primitive 1: formal invariants

Primitive 2: alignment anchors

Primitive 3: multi-objective constraints

Primitive 4: regression detection

Information-theoretic limits

A worked example

Use It

Ship It

Exercises

Key Terms

Further Reading

Code

Ships With