TAI BUI
Back to curriculum
LearnPython

Long-Running Background Agents: Durable Execution

Production long-horizon agents do not run in a bare `while True`. Every LLM call becomes an activity with checkpoint, retry, and replay. Temporal's OpenAI Agents SDK integration went GA in March 2026, and Claude Code Routines (Anthropic) runs scheduled Claude Code invocations without a persistent local process. Sessions pause on human input, survive deploys, and resume from the latest checkpoint keyed by `thread_id`. Underneath the new ergonomics is an old pattern, workflow orchestration, with one new input: LLM calls are non-deterministic activities that must be replayed deterministically on recovery.

TIME: ~60 minutesPREREQUISITES: Phase 15 · 10 (Permission modes), Phase 15 · 01 (Long-horizon agents)

Learning Objectives

  • Explain how replay lets a durable workflow skip activities that already completed.
  • Classify LLM calls, tool calls, and file writes as activities and keep workflow code deterministic.
  • Choose a checkpoint backend and a thread_id scheme for a long-running agent.
  • Decide when durable execution is the wrong tool, and pair it with fresh HITL and budget kill switches when it is right.

The Problem

Take an agent that runs for four hours. It calls three tools, prompts the user twice, and makes forty LLM calls. Halfway through, the host it is running on reboots. What happens?

  • In a naive while True loop, everything is lost. The run restarts from scratch, the three tool calls (with real side effects) execute again, the user is prompted again for things they already approved, and the forty LLM calls get re-billed.
  • With durable execution, the run resumes from the most recent checkpoint. Completed activities are not re-executed; their results come back from the durable log. The user does not re-approve anything, and the LLM calls already made are not re-billed.

Workflow engines have shipped this pattern for a decade (Temporal, Cadence, Uber's Cherami). What's new is that an LLM call is a kind of activity: non-deterministic, expensive, and full of side effects. It fits the pattern cleanly.

The backdrop is that long-horizon reliability decays. METR observes a "35-minute degradation", with success rate dropping roughly quadratically as the horizon grows. Durable execution lets you run longer than that reliability profile supports. Done right, that is a way to fail safely. Done wrong, it fails unsafely.

Warm-up0/3
1. A coding agent works through hundreds of steps on one task. Why does its success rate tend to fall as the run gets longer?Warm-up
A row of numbered steps 1 to 14 continuing to step N, with N in the hundreds, along an arrow labelled run gets longer. Below it a box reads chance the whole run still succeeds with a question mark.
Schematic of a long agent run. The success chance at the end is left as a question mark.
2. A permission mode asks for approval before each file write. What does that approval gate mainly control?Warm-up
A mock terminal showing an agent saying it wants to write a file, a prompt Allow this file write with yes and no, and the user answering y.
Typed illustration of a permission prompt.
3. A script sends 200 emails and crashes after email 120. If you rerun it from the top with no record of how far it got, what goes wrong?Warm-up
A bar of 200 emails: the first 120 marked sent, the last 80 not reached, with a crash line between them. An arrow shows the rerun beginning at email 1. A progress record box says none.
Setup only: what the stem describes.
Check: The Problem0/1
1. A four-hour agent run crashes near the end. Why does a naive retry cost far more than a durable retry?Explain
A four-hour run bar with the crash marked near its end and the counts 3 tool calls, 2 user prompts and 40 LLM calls. Below, two boxes, naive retry and durable retry, each with a cost marked by a question mark.
Setup only. The counts come from the lesson's worked example.

The Concept

Activities, workflows, and replay

  • Workflow: deterministic orchestration code. Defines the sequence of activities, the branches, the waits. Must be deterministic so it can be replayed from the event log without surprising divergence.
  • Activity: a non-deterministic, potentially failing unit of work. LLM call, tool call, file write, HTTP request. Each activity is logged with its inputs and (once complete) its outputs.
  • Event log: the durable backing store. Every activity start, complete, fail, retry, and every workflow decision is recorded.
  • Replay: on recovery, the workflow code re-runs from the start; every activity that already completed returns its logged result without re-executing. Only activities that had not completed are actually run.

This is the same shape as React re-rendering against a virtual DOM, or Git rebuilding a working tree from commits. Determinism in the orchestrator is what makes durability cheap.

Why LLM calls fit the pattern

LLM calls are:

  • Non-deterministic (temperature > 0; even temperature 0 drifts across model versions).
  • Expensive (money and latency).
  • Potentially failing (rate limits, timeouts).
  • Side-effectful (if they invoke tools).

That is the activity profile exactly. Wrapping each LLM call as an activity gives you retry with exponential backoff, checkpointing across restarts, and a replayable trace for debugging.

Checkpoints keyed by thread_id

LangGraph, Microsoft Agent Framework, Cloudflare Durable Objects, and Claude Code Routines all converged on the same API shape: a thread_id (or equivalent) identifies the session; each state transition persists to a backend (PostgreSQL default, SQLite for dev, Redis for cache); resume reads the latest checkpoint.

The backend choice matters:

  • PostgreSQL: durable, queryable, survives deploys. Default for LangGraph.
  • SQLite: local-dev only; loses data across hosts.
  • Redis: fast but ephemeral unless AOF/snapshot configured.
  • Cloudflare Durable Objects: transparently distributed; scoped by a unique key; survives for hours to weeks.

Human-input as a first-class state

Propose-then-commit (Lesson 15) requires a durable "waiting on human" state. The workflow pauses, the external queue holds the pending request, and an approval resumes from exactly that point. Without durability this is best-effort; with it, an overnight approval arrives and the workflow picks up in the morning.

The 35-minute degradation

METR observed that every agent class it measured shows reliability decay beyond ~35 minutes of continuous operation. Doubling the task duration roughly quadruples the failure rate. Durable execution does not fix this. It lets you run longer than the reliability profile supports. The safe pattern is to combine durability with checkpoints that require fresh HITL on re-entry, and with budget kill switches (Lesson 13) that cap total compute regardless of wall-clock time.

When durable execution is the wrong answer

  • Runs shorter than a few minutes with no human input. Overhead > benefit.
  • Strictly read-only information retrieval.
  • Tasks where correctness requires end-to-end within one context window (some reasoning tasks; some one-shot generation).
Check: The Concept0/21
1. In a durable execution system, what is a workflow?Define
An abridged Python function named workflow that calls fetch_docs, then call_llm, then write_report, with a simulated crash check after the first call.
An example of a workflow function, trimmed from the lesson's main.py.
2. What is an activity in a durable workflow?Define
The decorator and signature of the Python function fetch_docs from the lesson's main.py: decorated with @activity, taking a query string and returning an int, with the body left out.
An example of an activity: decorator and signature copied from the lesson's main.py.
3. What does the event log hold?Define
A workflow box connected to three activity boxes: fetch_docs, call_llm and write_report. A dashed bracket joining the three activities leads to a cylinder labelled event log, durable.json, whose contents are marked with a question mark.
Setup: where the event log sits in a run. Its contents are left unknown.
4. What is replay?Define
Attempt 1 shows fetch_docs and call_llm as run and write_report never started, then a crash with the event log kept. Attempt 2, recovery, is a box with a question mark.
Setup: a run that crashed with its log kept. The recovery step is left as a question mark.
5. What is a checkpoint in this lesson's sense?Define
Four states A to D in a row with an arrow between each pair. Under each arrow an orange marker labelled checkpoint. Below is a box reading what each checkpoint is with a question mark.
Setup: where checkpoints are written during a run. Their content is left as a question mark.
6. What does a thread_id do?Define
A two-line LangGraph style snippet: a config dictionary holding thread_id set to 7f3a9c, then graph.invoke called with inputs and config. Below it the text thread_id equals question mark.
Typed example, not captured. The id value is made up.
7. What is the 35-minute degradation?Define
A horizontal timeline of an agent run from 0 to 4 hours with a marker at 35 minutes. Below it a box reads the 35-minute degradation with a question mark.
Setup: the 35-minute mark on a run's timeline. The scale is illustrative.
8. What does non-determinism mean for a durable workflow?Define
A table of five expressions a workflow might read: os.getpid(), a short uuid, max of 4 and 9, len of hello times 3, and retries_so_far. A first-run column holds example values and a when-run-again column holds question marks.
Typed example values for illustration. The second column is unknown.
9. What must be true of workflow code so it can be replayed safely from the event log?Explain
A workflow code box connected to an event log cylinder that is kept after a crash, and below them a box asking for the required property of the workflow code with a question mark.
Setup: the workflow code and the event log it is replayed from.
10. Why does an LLM call fit the activity profile so well?Explain
A prompt box feeding an LLM call box, which produces a response box.
Setup: one LLM call inside an agent step.
11. A team needs checkpoints that survive deploys and can be queried later. Which backend does The Concept recommend as the durable default?Explain
A worker writing to a checkpoint store whose backend is marked with a question mark, and a new worker after a deploy reading from it. Two requirements are listed: survive deploys, and be queried later.
Setup: the team's two requirements. The backend is left open.
12. Why is SQLite described as local-dev only for checkpoints?Explain
A worker host containing an agent run and a checkpoints.db SQLite file on local disk. An arrow labelled later leads to a box reading resume the run on with a question mark.
Setup: where the SQLite checkpoint file lives. The resume location is left open.
13. According to The 35-minute degradation, what roughly happens to the failure rate when task duration doubles?Explain
A failure-rate axis with a solid bar labelled p for duration T, and only a question mark at duration 2T.
Setup: the bar at T is illustrative, not data. The value at 2T is the question.
14. An agent fetches one web page and summarises it in under a minute, with no human input. Should it be built on durable execution?Explain
Three boxes in a row: fetch one web page, summarise it, return summary. Notes say total time is under a minute and there is no human input.
Setup: the agent described in the stem.
15. Why does durable execution alone not make a six-hour run safe?Explain
A six-hour bar cut into ten 35-minute blocks and a final 10-minute block, with a note that the engine is durable so a crash does not lose the run, and a box asking why this is still not enough, marked by a question mark.
Setup: six hours is about ten and a half 35-minute blocks.
16. How does The Concept describe human approval inside a durable workflow?Explain
A workflow chain: edit files, ask for approval, deploy. Below the approval step, a dashed line notes that a person answers later, not right away. A box asks how the workflow treats this step, with a question mark.
Setup: a human approval step in the middle of a workflow.
17. A workflow reads the wall clock inside its decision code. After a crash, replay takes a different branch than the first run. What is the right fix?Analyze
A typed example where a branch is chosen from a clock reading. On the first run the clock reads t1 and branch A is taken. On replay it reads t2 and branch B is taken, although the log recorded branch A. A box asks for the right fix.
Typed illustration of the stem. t1, t2 and the cutoff are symbolic.
18. Checkpoints sit in a Redis instance with no AOF or snapshots. After a deploy restarts Redis, every long run starts from scratch. Which change addresses the cause?Analyze
Redis holding checkpoints, with no AOF and no snapshots, then a deploy restarts Redis, then a long run tries to resume. The result noted is that every long run starts from scratch. A box asks for the change that addresses the cause.
Setup: the situation in the stem.
19. Your team plans to run a 6-hour agent on durable execution to get past the 35-minute reliability decay. What else should the design include?Design
Three stacked layers: a 6-hour agent task, a durable execution layer with activities, event log, checkpoints and replay, and a box asking what else the design should include. A goal line says get past the 35-minute reliability decay.
Setup: the team's plan from the stem.
20. You must build an agent that runs six hours, edits files, and asks the user to approve a deploy that may wait until morning. Which design fits?Design
A six-hour bar for editing files, followed by edits done, ask user to approve deploy, and deploy. A note says the answer may not come until morning. A box asks which design fits.
Setup: the agent's requirements from the stem.
21. A coding agent lets one user run several sessions in parallel and resume any of them after a crash. Which thread_id scheme fits?Design
Three parallel sessions of one user, each linked to a box reading thread_id with a question mark and a note that each may crash and resume.
Setup: one user with several sessions. Three are shown for illustration.

Use It

code/main.py is a minimal durable-execution engine in stdlib Python. It has:

  • An EventLog backed by a JSON file.
  • An @activity decorator. It looks for a completed result in the log and returns it if found. Otherwise it appends a started event, runs the function, and appends a done event with the result.
  • A workflow function that sequences three activities (fetch_docs, call_llm, write_report), with an optional simulated crash.

Replay here is just the workflow running again against the same log, with the decorator answering from it. The driver runs the workflow twice over. In the naive run the log is wiped between attempts. In the durable run it is kept. Attempt 1 crashes after call_llm in both. The naive retry runs all three activities again, while the durable retry replays the first two from the log and runs only write_report.

Check: Use It0/10
1. Why does the activity wrapper append a started event before running the function?Explain
The wrapper function from main.py with the log lookup omitted. It appends a started event, runs the function, appends a done event and returns the result. The started append is highlighted.
Adapted from the lesson's main.py with the print lines and the lookup removed.
2. In main.py, how does the toy engine decide that an activity has already completed?Apply
Attempt 2 after a crash: the workflow calls fetch_docs with the query hello, the call passes through the activity wrapper, which consults a log file. A box asks how the wrapper decides the activity already completed, with a question mark.
Setup: the lookup the stem asks about.
3. A run crashes after fetch_docs and call_llm finish but before write_report starts, and the event log is kept. Which activities execute on retry?Apply
Three activities in a row: fetch_docs and call_llm finished with results in the log, write_report not started. A crash note says the event log is kept. A box asks which activities execute on retry, with a question mark.
Setup: the state at the moment of the crash.
4. In main.py, the process dies after the started event for an activity is written but before its done event. What happens on retry?Apply
An event log table: fetch_docs done with result 15, call_llm done with result summary(15_docs), and write_report with status started and no result. A note says the process dies after the last row and before a done row. A box asks what happens to write_report on retry.
Typed, not captured: rows follow the log format of main.py, and the crash point is the one in the stem.
5. In main.py, fetch_docs returns len(query) * 3. What does it return for the query 'hello'?Apply
The fetch_docs function from main.py returning len(query) times 3, and below it the call fetch_docs with the query hello, whose result is a question mark.
Code copied from the lesson's main.py.
6. Run the durable workflow with crash_after=2 on the first attempt, keep the log, and let attempt 2 finish. What does count_runs report at the end?Apply
Attempt 1 runs fetch_docs and call_llm and then crashes. Attempt 2 has three question-mark boxes. A box asks for the value of count_runs on the log at the end.
Setup: the scenario in the stem. Outcomes of attempt 2 are left open.
7. A workflow completes on query 'hello' with no crash. What value does it return?Apply
The three activity bodies from main.py (fetch_docs returns len(query) times 3, call_llm returns summary of doc_count docs, write_report returns report:// followed by the summary), and a chain from the input hello through the three activities to a question mark.
Code copied from the lesson's main.py.
8. A run on query 'hello' has finished and its log is kept. Then fetch_docs is called with the query 'hi' against the same log. What happens?Apply
The real event log of a finished run on hello: a started and a done row for each of fetch_docs, call_llm and write_report. Below it the call fetch_docs with the query hi against this log, with a question mark for the outcome.
Log rows captured by running the lesson's main.py with no crash.
9. A write_report activity sends an email, then the process dies before its done event is written. On retry the recipient gets a second email. What went wrong in the design?Analyze
Attempt 1 of write_report: a started row, an email sent, then the process dies with no done row written. Attempt 2: a started row, an email sent, a done row. The recipient's inbox holds 2 copies. A box asks what went wrong in the design.
Setup: the sequence in the stem.
10. Two concurrent sessions share the toy engine and call fetch_docs with the same arguments. One session gets the other's result. Which missing piece caused it?Analyze
Two sessions each call fetch_docs with the query hello against one toy engine with one log file. Session B ends up with the other session's result. A box asks which piece is missing.
Setup: the shared-engine situation from the stem.

Ship It

outputs/skill-durable-execution-review.md reviews a proposed long-running agent deployment for durable-execution shape: activities, determinism, checkpoint backend, human-input state, and HITL-on-resume policy.

Author's Note

I'd run main.py first and read the two halves of its output side by side. Both start with an attempt that crashes after call_llm. The naive half wipes the log between attempts and starts all three activities again. The durable half keeps the log, so the retry replays fetch_docs and call_llm from it and only write_report runs.

The detail I'd study is where the log gets written. The wrapper appends a started event before the function runs and a done event after, and lookups only match done. A crash in the middle of an activity leaves a started with no result, so that activity runs again on retry. Any activity with side effects needs to be safe to repeat, or needs a key the other side can dedupe on.

I'd also keep the toy in proportion. It matches cached results by activity name and arguments, so two calls with identical arguments would share one result. It has no thread_id, no backoff on retry, and no human-input wait. Treat it as the shape of replay, and reach for a real engine for the rest.

Exercises

  1. Run code/main.py. Observe the difference in activity-execution count between naive retry and replay. Change the crash point and show the replay count changes accordingly.

  2. Convert the toy engine to use thread_id explicitly. Simulate two concurrent sessions sharing the engine and confirm their event logs do not collide.

  3. Take one activity in the toy engine. Introduce a non-determinism (a wall-clock timestamp inside a workflow decision). Demonstrate the divergence on replay. Explain how real engines handle this (side-effect registration, Workflow.now() APIs).

  4. Read the LangChain "Runtime behind production deep agents" post. List every state that the runtime persists and name which failure mode each covers.

  5. Design a checkpoint policy for a 6-hour autonomous coding task. Where do you checkpoint? What does resume-on-crash look like? What requires fresh HITL?

Key Terms

TermWhat people sayWhat it actually means
Workflow"Agent's script"Deterministic orchestration code; replayable from event log
Activity"A step"Non-deterministic unit (LLM call, tool call); logged before and after
Event log"The backing store"Durable record of every state transition
Replay"Resume"Re-run workflow; completed activities return logged results without re-execution
Checkpoint"Save point"Persisted state keyed by thread_id; latest-wins on resume
thread_id"Session key"Identifier that scopes durable state
35-minute degradation"Reliability decay"METR: success rate drops ~quadratically with horizon
Non-determinism"Drift on replay"Wall clock, random, LLM output; must be registered as side effect

Further Reading

Code

Ships With