LearnPython

Checkpoints and Rollback

Persist each graph-state transition. When a worker crashes, its lease expires and another worker resumes from the latest checkpoint. Cloudflare Durable Objects can hold that state for hours or weeks, and propose-then-commit (Lesson 15) supplies a rollback plan per action. Post-action verification closes the loop. EU AI Act Article 14 requires effective human oversight for high-risk systems, which in practice means checkpoints an auditor can query, rollbacks you have rehearsed, and an audit trail that survives a deploy. The sharp failure mode: without idempotency keys and precondition checks, a retry after a transient failure can double-execute an approved action, and post-action verification is what catches it.

TIME: ~60 minutesPREREQUISITES: Phase 15 · 12 (Durable execution), Phase 15 · 15 (Propose-then-commit)

Learning Objectives

Explain why an idempotency key alone cannot stop an overdraft, and add the precondition check that closes the gap.
Choose in-band, compensating, or out-of-band rollback for a given action, and state when an action with no rollback needs stronger human review.
Design post-action verification that re-reads the target state instead of trusting a 200 response.
Trace the double-execute failure through a crash and resume, and order the status writes to avoid it.
Run `main.py` and point out where its "committed"-before-execute ordering leaves a durability gap.

The Problem

Durable execution (Lesson 12) makes a crashed agent resumable. Propose-then-commit (Lesson 15) makes an approved action auditable. This lesson joins them and asks the awkward question: what happens when an approved action executes partway, crashes, and resumes? When does the rollback run, and against what state?

Real systems wire this up differently:

LangGraph checkpoints every graph-state transition to PostgreSQL. On worker crash, the lease releases and another worker resumes at the latest checkpoint. Workflows pause on interrupt(), which itself persists.
Cloudflare Durable Objects hold per-key state across hours or weeks. Co-locate the computation with the storage for the approved action.
Microsoft Agent Framework exposes Checkpoint primitives in the workflow API, and replay plus idempotency covers retries.

The combination that works in each case has four parts: an idempotency key (prevents double-execute), a precondition check (state is still what we approved against), a post-action verify (the side effect happened), and rollback when verify fails.

Warm-up0/3

1. A durable workflow resumes after its host reboots mid-run. What happens to the activities that already completed?Warm-up

A workflow with activities 1 and 2 completed and activity 3 running when the host reboots, then three boxes marked with a question mark for what happens to each activity after restart. — Schematic of the situation in the question.

2. In propose-then-commit, what does the idempotency key stored with a proposal protect against?Warm-up

A proposal record with fields action and idempotency_key k, followed by a question mark for what the stored key protects against. — Typed schematic of a proposal record.

3. What is the documented mitigation for rubber-stamp approvals in propose-then-commit?Warm-up

Four proposals in a queue, each with an Approve clicked label, followed by a question mark for the documented mitigation for rubber-stamp approvals. — Illustrative queue, not real data.

The Concept

Persist every transition

A graph-state transition is any step that moves the workflow from one named state to another. Naive implementations persist only at specific commit points. Production implementations persist each transition. A few extra writes cost little next to what you get back: replay can land anywhere, and lease recovery is precise.

Lease recovery

When a worker crashes, the workflow is not lost. The lease (a short-lived claim that this worker is executing this run) expires, and another worker picks up the latest checkpoint and resumes. This is what lets production systems survive rolling deploys without losing in-flight work.

Idempotency plus preconditions

Idempotency alone is not enough. Say a workflow is approved to "transfer $100 from A to B when balance > $1000." It commits, crashes mid-execution, and resumes. If the resume only checks the idempotency key, the transfer runs once, which is correct. Now suppose A's balance drops to $500 through a different workflow between the crash and the resume. The idempotency check still passes, but the precondition does not, and without a precondition check you ship an overdraft.

Consequential actions need both:

Idempotency key: prevents double-execute.
Precondition check: confirms the state is still consistent with what was approved.

Post-action verification

"The tool returned 200" is not verification. Real verification re-reads the target state and confirms the side effect happened. Patterns:

Database update: UPDATE ... RETURNING *, then assert the returned row matches intended state.
Email send: check the sent folder for the message ID after submission.
File write: read the file back and hash it.
API call: follow-up GET on the target resource.

If verify fails, the workflow is in a known-bad state and rollback engages.

Rollback plans

Each consequential action in propose-then-commit (Lesson 15) carries a rollback plan. There are three types:

In-band rollback: reverse the side effect directly (DELETE after INSERT, a correction email after a send).
Compensating transaction: a new action that neutralizes the original (the standard SAGA pattern).
Out-of-band rollback: alert a human, pause the workflow, and leave the bad state for investigation.

If an action cannot be undone, the proposal has to say so. Actions with no rollback need stronger HITL at commit time (Lesson 15 challenge-and-response).

EU AI Act Article 14, operationally

Article 14 requires "effective human oversight" for high-risk systems. Implementers read that as:

Checkpoints an auditor can query.
Rollbacks that have been rehearsed, meaning tested end-to-end at least once.
An audit trail that survives a deploy, so the checkpoint backend cannot be ephemeral.
Failed verifications that raise an alert instead of landing silently in a log.

A workflow that crashes mid-commit, resumes, and completes the side effect with no verify and rollback path does not pass this test.

The sharp failure mode: the double-execute

The most common production incident in this space goes like this:

Action approved, idempotency key k.
Commit starts, executes, returns 200.
Workflow crashes before persisting the "committed" status.
Workflow resumes, sees "approved but not committed", and re-executes.
The side effect fires twice.

The mitigation: persist an "in-flight" intent before execution, execute with an idempotency key, and mark "committed" only after post-action verification succeeds. If the action fires and the status write fails, you know to verify and re-fire if needed. If the status write succeeds and the action fails, you verify and fire once via the recovery path.

Check: The Concept0/27

1. What is a checkpoint in a durable agent workflow?Define

Three workflow states in a row named new, committed and verified, and a question mark for what a checkpoint is. — Typed schematic. State names are the ones used in main.py.

2. What is a lease on a workflow run?Define

Typed schematic.

3. What does a precondition check assert before an approved action fires?Define

A pipeline: Approved proposal, a dashed box labelled check with a question mark, then the action fires its side effect. — Typed schematic.

4. What is post-action verify?Define

A pipeline Approved, Execute, then a dashed box post-action verify with the sub label does what and a question mark. — Typed schematic.

5. What is an in-band rollback?Define

A table with two existing rows and one changed row written by an action, outlined in red, followed by a question mark for what an in-band rollback is. — Typed schematic.

6. What is a compensating transaction?Define

Typed schematic using the lesson's transfer example.

7. What does the mark-as-done-first pattern do?Define

A checkpoint file holding a status record and a destination DB holding balances, followed by a question mark for what the mark-as-done-first pattern does. — Typed schematic. Names are taken from main.py.

8. Which of these is NOT one of the three rollback types?Define

A proposal record for transferring $100 from A to B with an idempotency key and a rollback_plan field filled with a question mark. — Typed schematic.

9. Which set matches the operational reading of EU AI Act Article 14?Define

A box quoting Article 14's phrase effective human oversight for high-risk systems, with one wide dashed question mark box for the operational reading. — The quoted phrase is from the lesson.

10. In the $100 transfer example, why does an idempotency key alone fail to prevent an overdraft?Explain

A pipeline Commit with key k saved, Crash mid-execution, Resume with key k checked, and a dashed Overdraft box with a question mark, under the approval text transfer $100 from A to B when balance is over $1000. — Typed schematic using the lesson's transfer example.

11. A worker crashes mid-run. In lease recovery, what happens next?Explain

Four workflow states where states 1 and 2 are done, Worker A crashes in state 3, and state 4 is not reached, followed by the question what happens next. — Typed schematic.

12. A consequential action genuinely cannot be undone. What must its proposal do?Explain

Typed schematic.

13. What causes a side effect to fire twice in the double-execute failure?Explain

A target system log with two identical lines effect applied, followed by the question which cause produces a double execution. — Typed schematic.

14. Why persist every graph-state transition instead of only specific commit points?Explain

Six steps in a row. The commit-points-only lane has writes at steps 3 and 6, and the every-transition lane has a write at all six steps. The question asks why persist every transition. — Illustrative: which steps count as commit points is made up.

15. Which of these counts as real post-action verification for an email send?Explain

An agent calling send_email through a mail API toward a mail system, followed by the question which check counts as real post-action verification. — Typed schematic.

16. What is the mitigation for the double-execute failure?Explain

Three boxes: action approved with idempotency key k, the action runs and the workflow later resumes, the side effect fires twice, followed by the question what the mitigation is. — Typed schematic.

17. How does a compensating transaction differ from an in-band rollback?Explain

Two columns titled In-band rollback and Compensating transaction, each with a dashed box asking how it undoes the transfer of $100 from A to B. — Typed schematic using the lesson's transfer example.

18. Why does the Article 14 reading rule out an in-memory checkpoint backend?Explain

An in-memory checkpoint store holding cp 1 to cp 3, and a question mark for why the Article 14 reading rules this backend out. — Typed schematic.

19. Which failure does a precondition check catch that an idempotency key does not?Explain

Four boxes in a row: Approved proposal, Idempotency key gate, Precondition gate and Execute side effect, followed by the question which failure the precondition catches that the key does not. — Typed schematic.

20. Why does a failed verify mean rollback has to engage?Explain

Execute, then Verify marked red because the re-read fails, then one dashed question mark box for what comes then. — Typed schematic.

21. A transfer is approved 'when balance > $1000'. After a crash, A's balance is $500 and the resume passes the idempotency check. What should a workflow with both checks do?Apply

Two checks on resume: the idempotency key check passes, and a precondition check where A holds $500 while the approval needed a balance over $1000, followed by the question what the workflow should do. — Typed schematic using the stem's figures.

22. A job must write a config file to a server and confirm the write worked. Which verification fits the lesson?Apply

A job writing a config file to a server, followed by the question which verification confirms the write worked. — Typed schematic.

23. An agent approved to email a refund notice crashes after the send call returns 200 but before saving its status. On resume it sees "approved but not committed". What is the safest next step?Analyze

Refund email flow: approved, the send call returns 200, the workflow crashes before saving its status, and on resume it sees approved but not committed, followed by the question what the safest next step is. — Typed schematic of the stem.

24. A workflow marks a profile update committed after the API returned 200. Weeks later a customer says the profile never changed. Which ingredient was missing?Analyze

Typed schematic of the stem.

25. Verify fails at 3 a.m. The workflow writes a log line and stops, and the wrong balance sits there until Monday. Which Article 14 expectation was missed?Analyze

Timeline: at 3 a.m. verify fails, the workflow writes a log line and stops, and on Monday the wrong balance is still there, followed by the question which Article 14 expectation was missed. — Typed schematic of the stem.

26. Team A persists state only at commit points, and Team B persists every transition. A worker dies between two non-commit steps. Which statement is right?Analyze

Six steps with steps 3 and 6 highlighted as commit points and a worker dying between steps 4 and 5, then two question mark boxes for where Team A and Team B resume. — Illustrative: which steps are commit points is made up for the picture.

27. An agent proposes wiring funds to an external account, and no rollback exists. Which design fits the lesson?Design

An agent proposes a wire through a bank to an external account, with a note that no rollback exists, followed by the question which design fits the lesson. — Typed schematic.

Use It

code/main.py implements a checkpointed workflow with idempotency, preconditions, verify, and rollback. The driver simulates four scenarios: clean run, retry after crash (idempotency catches), precondition fail (workflow aborts without firing), verify fail (rollback fires).

Check: Use It0/12

1. Why does `Checkpoint.save` write a temp file, fsync, and rename instead of overwriting the file?Explain

The Checkpoint.save method from main.py: load the data, set the key, write it to a temp path, flush and fsync, then os.replace the temp file onto the checkpoint path. — Code copied from main.py with its comments removed.

2. In `run_transfer`, why does every terminal status return before the precondition is checked?Explain

The first lines of run_transfer: the checkpoint lookup, the terminal_results mapping for four statuses with its values elided, a return for any terminal status, and then the balance check against min_balance. — Code copied from main.py with its comments removed and the mapping values elided.

3. In `main.py` Scenario 2, the transfer runs, the process crashes, and the same `txid` is retried. Why does `balance_A` drop only once?Apply

Scenario 2 timeline: run_transfer for tx-002, the transfer applied taking balance_A from 1400 to 1300, a crash, and a retry with the same txid, followed by the question why balance_A dropped only once. — Values from running main.py.

4. Scenario 1 runs `run_transfer` for tx-001, amount 100, min_balance 200, on a DB of A=1500 and B=200. What does it print?Apply

The starting DB with balance_A 1500 and balance_B 200, the run_transfer call for tx-001 with amount 100 and min_balance 200, and a result marked with a question mark. — Setup typed from the question; main.py's DB starts with these values.

5. In Scenario 2, the crash comes after `persist_transfer` and the same txid is retried. What does the retry return, and where does balance_A end up?Apply

Scenario 2 setup: balances 1400 and 300 at the start, a run of tx-002 that crashes after the transfer, and a retry of the same txid whose result and balance_A are question marks. — Start balances come from Scenario 1's real output.

6. After Scenario 2, balance_A is 1300. Scenario 3 requests a transfer of 10,000 with min_balance 200. What happens?Apply

Scenario 3 setup: balance_A is 1300 and run_transfer is called for tx-003 with amount 10,000 and min_balance 200, with the printed result a question mark. — Setup typed from the question.

7. balance_A is 300 and min_balance is 200. A new txid requests a transfer of 150. What does `run_transfer` return?Apply

balance_A is 300 and min_balance is 200, run_transfer is called with a new txid and amount 150, and the result is a question mark. — Setup typed from the question; the txid is a placeholder.

8. Apart from `inject_verify_fail`, what makes `run_transfer` roll the transfer back?Apply

Lines from run_transfer after persist_transfer: an if statement with inject_verify_fail or three question marks guarding rollback_transfer, a rolled-back save and a return of verify-fail-rolled-back. — Code copied from main.py with the second condition masked.

9. Scenario 4 forces a verify failure on tx-004 and compares balances before and after. What does `balances_before_after_equal` show, and why?Apply

Scenario 4 code: copy the DB, run tx-004 for 100 with inject_verify_fail set, copy the DB again, and print whether the two copies are equal, shown as a question mark. — Code copied from main.py.

10. Suppose the `committed` save in `run_transfer` is moved to after the injected crash line, and Scenario 2's crash and retry run again. What happens?Apply

run_transfer edited so persist_transfer and the injected crash come first and the cp.save of status committed comes after them, with Scenario 2 about to run again. — Edited copy of main.py code, as the question describes.

11. A crash hits right after `cp.save` records `committed` and before `persist_transfer` runs. The retry returns idempotent-skip. What is true now?Analyze

Three lines of run_transfer: cp.save with status committed, a crash marker, and persist_transfer, followed by the note that the retry returns idempotent-skip and the question what is true now. — Code lines from main.py with the crash position marked.

12. You must close the `committed`-before-execute gap in `main.py` without losing retry safety. Which change fits the lesson?Design

Three lines of run_transfer with a crash marker between cp.save of status committed and persist_transfer, followed by the question which change closes the gap. — Code lines from main.py with the crash position marked.

Ship It

outputs/skill-rollback-rehearsal.md designs a rollback-rehearsal test for a proposed workflow and audits the checkpoint backend for audit-trail persistence.

Author's Note

I'd run main.py first and read the four scenario outputs before the prose. Scenario 2 crashes after the transfer and then retries the same txid. The retry returns idempotent-skip, and balance_A only drops by 100 (1400 to 1300), so tx-002 ran once, not twice. Scenario 4 compares the balances before and after a forced verify failure, and they come back equal.

The part I'd read slowly is the order of writes in run_transfer. The code saves committed to the checkpoint before it calls persist_transfer, and only promotes the record to verified after the read-back. That is not the mitigation described above, which marks "committed" after verification. Here committed is doing the job of the in-flight intent. It works, but there is a gap, and the comment in the code says so: a crash between the save and persist_transfer leaves a committed record for a transfer that never ran, and the retry skips it. Closing that gap means carrying the idempotency key into the destination write or checking the destination before trusting the marker.

Two smaller things I'd notice. The precondition checks the balance that would remain after the transfer against min_balance, which is a different gate from the "balance > $1000" wording in the example. It also runs only after the terminal-state lookup, so a retry of a finished txid never reaches it. And Checkpoint.save writes a temp file, fsyncs, then renames, so a crash mid-write leaves the previous record intact. That is a small version of the durable backend the Article 14 reading above calls for.

For Exercise 2 I'd expect the duplicate to show up once the committed save moves past the injected crash (after the verify step). The crash then lands before any record exists, the retry sees status new, the precondition still passes (1300 - 100 >= 200), and the transfer fires a second time.

Exercises

Run code/main.py. Verify the four scenarios. For the crash-during-commit case, confirm the action fires once across retries.
Modify the "mark as done first, then do it" pattern so the status write fires after the action. Rerun the crash scenario. Measure how many duplicate actions fire.
Design a rollback plan for a specific production action (e.g., "post to a Slack channel"). Classify as in-band, compensating, or out-of-band. Justify the choice.
Take one workflow you know. Identify every state transition. Mark each with a durability requirement (persist / do not persist). Count the ones you are currently not persisting.
Rehearsed-rollback test: design an end-to-end test that runs a real workflow, crashes it, and confirms the rollback path fires. What does the test assert?

Key Terms

Term	What people say	What it actually means
Checkpoint	"Save point"	Every graph-state transition persists to a durable store
Lease	"Worker claim"	Short-lived claim that a worker is executing a run; expires on crash
Precondition	"State gate"	Assertion that the state is still consistent with the approved action
Post-action verify	"Re-read check"	Confirm the side effect actually happened in the target system
In-band rollback	"Direct undo"	Reverse the side effect with the inverse operation
Compensating transaction	"SAGA undo"	A new action that neutralizes the original
Mark-as-done-first	"Status write order"	Persist the committed status before returning from commit
Article 14	"EU AI Act human oversight"	Operational: queryable checkpoints, rehearsed rollbacks, auditable trail