Production Scaling — Queues, Checkpoints, Durability

A prototype multi-agent system runs fine on one laptop with three agents in an in-memory event loop. Then you move to production:

• Agents sometimes run for hours (long research, human-in-the-loop waits).
• Worker processes crash, and a restart loses state.
• Peak load is 10x average, so you need horizontal scaling.
• Users pay per agent-run, so charging has to be exactly-once.

An in-memory event loop handles none of this. You need a durable execution layer underneath. The 2026 options are:

1. A workflow engine with checkpoints (Temporal, LangGraph runtime).
2. A message queue with a state store (Postgres + SQS/RabbitMQ).
3. Actor-model frameworks (MegaAgent's producer-consumer per agent).
4. Hand-rolled FastAPI + Postgres (Bedi's argument).

The code in this lesson covers the core of the first option (a checkpoint log with crash resume) and the shape of the third (a per-agent state machine). It also measures async against threads. The other options we compare on paper.

Durable execution, the pattern

A durable-execution engine persists the full program state after each "step" (a super-step, in LangGraph's language). When a worker crashes:

worker crashes mid-step
  -> lease timeout
  -> another worker picks up the thread_id
  -> resumes from last checkpoint
  -> no duplicate side effects

Three requirements make this work:

• Serializable state. Everything the agent knows has to be persistable. A closure holding a live database connection does not survive a restart.
• Deterministic resume. Given the same state and inputs, the agent produces the same actions, or defers to an external deterministic oracle for LLM calls.
• Idempotent side effects. External calls (tool calls, payments) must be idempotent or carry a deduplication key.

LangGraph writes a checkpoint after each super-step, Temporal writes after each activity, and Restate uses event-sourced journals. All three implement the same pattern.

LangGraph's runtime

Each agent has a thread_id, state is a typed dict, and each super-step writes a row to the checkpoints table. On resume, the runtime replays from the last checkpoint instead of from scratch. An agent can call interrupt() to wait for human input: the runtime persists the state and releases the worker, and when input arrives, any worker can pick the thread back up.

This is the reference production design as of April 2026.

MegaAgent's per-agent queue

arXiv:2408.09955 describes a scale experiment with thousands of concurrent agents in one cluster. The architecture:

agent i:
  state ∈ {Idle, Processing, Response}
  in_queue   <- messages addressed to agent i
  out_queue  -> replies + side effects

coordinators:
  intra-group chat  (agents in the same group)
  inter-group admin chat  (high-level routing)

Two layers of coordination let conversation inside a group stay dense while traffic between groups stays sparse. That split is how the system keeps cost roughly linear as agent count grows.

Async vs thread-per-job

LLM calls are I/O-bound. A thread waiting on the next token is idle about 99% of the time, and threads cost ~1MB of RAM each, so 10,000 concurrent calls means 10GB of stacks before any work happens.

Fibers (Python asyncio, Go goroutines, Rust tokio) yield cooperatively on I/O, so the same 10,000 calls fit comfortably in one process. At LLM-agent scale, async is the architecture itself, not a tuning step.

CPU-bound post-processing (embedding, tokenizer tricks) still wants threads or processes. Keep the I/O layer separate from the CPU layer.

Bedi's counterpoint

"Scaling Agentic Software" (Ashpreet Bedi, 2026) argues that most teams over-engineer before they have measured load. His default:

• FastAPI + Postgres.
• Each agent run is a row, updated in place with optimistic concurrency.
• Background jobs via pg_notify or a simple Celery worker.
• Retry policy in application code.

For loads under roughly 100 concurrent agent-runs on manageable tasks, that is often enough. Upgrade when you measure it failing.

Adopt a durable-execution framework when you hit a concrete problem a simple architecture can't solve. Adopting one earlier costs you ceremony that doesn't pay off yet.

Exactly-once semantics

For paid agent runs you want "exactly-once effective", which means at-least-once delivery plus an idempotent consumer. Three moves get you there:

• Dedup key per run. Include it in every side-effect call.
• Outbox pattern. Side effects are written to a table first, and a separate process executes them. Both steps are idempotent.
• Compensating transactions. When a side effect succeeds but the write that tracks it fails, schedule a compensation.

These are database-engineering patterns, not LLM-specific ones. The only LLM tax is that the calls are slow. The rest is standard distributed systems.

Rainbow deployment

Anthropic's multi-agent research system uses "rainbow deployments": several versions of the agent runtime run concurrently, so long-running agents don't get killed on every code deploy. You canary a new version on a slice of traffic and retire the old version once its agents finish.

This is standard for long-running stateful systems. What changes in 2026 is that agents can live for hours, so the deploy cycle has to wait for them.

The production checklist

• Durable state (checkpoints, snapshots, or outbox + replayable log).
• Idempotent side effects.
• Async I/O layer for LLM calls.
• At-least-once delivery with dedup.
• Rainbow/canary deployment for stateful workloads.
• Observability: per-agent traces, super-step audit, retry counter.

main.py implements:

• CheckpointStore: a SQLite-backed checkpoint log keyed by thread_id. Each super-step writes a row.
• run_agent_with_checkpoint(store, thread_id, ...): runs a tiny agent that counts up to a goal, can simulate a crash at a chosen super-step, and on the next call resumes from the last checkpoint.
• AgentQueue: a per-agent Idle / Processing / Response state machine with an in-queue and an out-queue. It lives in memory and advances one transition per step() call.
• demo_async_vs_threads(): runs 500 simulated "LLM calls" (a 50 ms sleep each) with asyncio and with threads, and prints the wall-clock time for each plus the ratio.

Run:

python3 main.py

What you should see: the first worker crashes at super_step 3, the last checkpoint holds counter 3 at super_step 2, and the second worker resumes and finishes at 5. The queue demo walks one agent through two messages. The benchmark prints async finishing close to the 50 ms sleep, with threads slower by a small multiple on my run. It does not measure memory. The ~1MB per thread stack in the printed ratio line is a stated figure, not a measurement.

outputs/skill-scaling-advisor.md helps you choose a durable-execution approach: FastAPI + Postgres, LangGraph runtime, Temporal, or a custom build. It weighs load, how long state has to be retained, and how often you deploy.

Production hardening:

• Start simple (Bedi's rule). FastAPI + Postgres until you measure it failing.
• Instrument before you optimize. Per-run latency histogram, per-step time, retry count, failure categories.
• Use an outbox for side effects, especially payments and external API calls.
• Deploy rainbow-style so in-flight agent runs survive a deploy.
• Adopt a durable-execution engine (Temporal / LangGraph / Restate) when you hit a specific problem: hour-long human-in-the-loop waits, cross-region coordination, or complex retry and compensation policies.
• Keep the I/O layer async. Reserve threads for CPU-bound post-processing.

1. Run main.py. Confirm checkpoint resume works; measure async vs thread concurrency difference.
2. Implement an outbox table: every tool call writes to outbox first, then a separate async task executes. Verify idempotency by running the tool call twice.
3. Simulate a rainbow deploy: two concurrent runtime versions; route half of new thread_ids to each; confirm that in-flight threads on the old version are not interrupted.
4. Read LangGraph's runtime doc (linked below). Identify which features of the runtime would take the longest to replicate in a hand-rolled FastAPI + Postgres version. Is that a reason to adopt, or can you defer?
5. Read MegaAgent (arXiv:2408.09955) Section 3. The two-layer coordination (intra-group + inter-group admin chat) is explicit. Sketch how you would map this to a message queue with two queue families.

• LangChain: The runtime behind production deep agents: LangGraph runtime design
• MegaAgent: per-agent producer-consumer queue; two-layer coordination at thousands of concurrent agents
• Matrix: decentralized framework with message queues as the coordination substrate
• Temporal docs: the reference workflow engine for durable execution
• Anthropic: Multi-agent research system: production lessons including rainbow deployment