LearnPython

Cold Start Mitigation for Serverless LLMs

A 20 GB model image takes 5-10 minutes (7B) to 20+ minutes (70B) to go from cold to serving. On a serverless endpoint that is an outage, not a warm-up. Five layers attack it: pre-seeded node images (Bottlerocket on AWS, dual-volume arch), model streaming (NVIDIA Run:ai Model Streamer, native in vLLM), GPU memory snapshots (Modal checkpoints, up to 10x faster restart), warm pools (`min_workers=1`), and tiered loading (ServerlessLLM's NVMe to DRAM to HBM pipeline, 10-200x latency reduction). A sixth pattern, live migration, moves input tokens (KB) instead of KV cache (GB). Modal publishes 2-4s cold starts as a floor and Baseten 5-10s by default, sub-second with pre-warming. You will measure each phase, budget it, and stack the layers that fit your SLA.

TIME: ~60 minutesPREREQUISITES: Phase 17 · 02 (Inference Platform Economics), Phase 17 · 03 (GPU Autoscaling)

Learning Objectives

Enumerate the five layers of cold-start mitigation and name one tool or pattern at each layer.
Compute total cold-start time as a sum of (node provision) + (weights download) + (weights load into HBM) + (engine init) for a 70B model.
Explain why live migration transfers input tokens (KB) not KV cache (GB) and what the penalty is (recomputation).
Name the warm-pool trade-off (pay for idle GPU or accept cold-start tail) and the SLA threshold at which `min_workers > 0` becomes mandatory.

The Problem

Your serverless LLM endpoint scales to zero overnight. At 8 a.m. traffic spikes, and the first request waits through four steps:

Karpenter provisions a GPU node: 45-60s.
The container pulls a 30 GB image with weights: 120-300s.
The engine loads weights into HBM: 45-120s depending on model size and storage speed.
vLLM or TRT-LLM initializes CUDA graphs, the KV cache pool, and the tokenizer: 10-30s.

That totals 220-510s (roughly 3-8 minutes) before one token comes back, against an SLA of 2s. So you ship a warm pool (min_workers=1) and the problem seems to vanish. Now you pay for one idle GPU around the clock. Give 5 products one warm replica each and that is 5 × 24 × 30 = 3,600 GPU-hours/month, whether or not a single user called.

Cold-start mitigation is how you keep serverless economics while getting close to the latency of an always-on service.

Warm-up0/3

1. A serverless GPU endpoint scales to zero overnight to save money. What do you pay for that choice when traffic returns in the morning?Warm-up

A timeline from an idle overnight period with no replica running to 8 a.m. when traffic returns and a first request arrives, asking what that request pays for. — Setup only; the scale-to-zero overnight gap is stated directly in the question.

2. An API must answer within 2 seconds, but a new server takes several minutes to boot. What is the standard way to protect latency, and what does it cost?Warm-up

A 2 second latency SLA next to a new server that takes several minutes to boot, with a question about the standard way to protect the SLA and what it costs. — Everyday setup, no specific vendor or platform.

3. A 30 GB container image must run on a fresh machine. Which setup gets the container running soonest?Warm-up

A 30 gigabyte container image that must run on a machine that just booted, with a question about which starting condition for that machine gets the container running soonest. — Everyday setup; no specific starting condition is shown.

Check: The Problem0/3

1. What is a cold start for an LLM endpoint?Define

A term card showing Cold start, tagged as a term for an LLM endpoint, with a question asking what it is. — The term as the question names it; its definition is the question.

2. What does scale-to-zero mean for a serverless endpoint?Define

A term card showing Scale-to-zero, tagged as a serverless endpoint behavior, with a question asking what it means. — The term as the question names it; its definition is the question.

3. Five products each keep one warm replica running 24x7 for a 30-day month. How many GPU-hours is that, whether or not a user calls?Apply

A table of four quantities, 5 products, 1 replica per product, 24 hours a day and 30 days in the month, asking for the total GPU-hours for the month. — All four quantities taken directly from the question; no total is shown.

The Concept

Layer 1: pre-seeded node images (Bottlerocket)

On AWS, Bottlerocket's dual-volume architecture separates OS from data. Snapshot the data volume with your container image pre-pulled; reference the snapshot ID in your EC2NodeClass. New nodes boot with weights already on local NVMe, so step 2 and part of step 3 vanish. It works with Karpenter natively. Typical savings: 2-4 minutes per cold start for large models.

Equivalent on GCP: custom VM images with pre-baked container layers. On Azure: managed disk snapshots with the same pattern.

Layer 2: model streaming (Run:ai Model Streamer)

Don't load the full file before touching the first request. Stream weights into GPU memory layer by layer and start processing once the first transformer block is resident. The NVIDIA Run:ai Model Streamer ships native in vLLM 2026 and reads from S3, GCS, and local NVMe. It cuts weight-load time roughly in half for large models by overlapping I/O with compute setup.

Layer 3: GPU memory snapshots (Modal)

Modal takes a checkpoint of the GPU state (weights, CUDA graphs, KV cache region) after first load. Later restarts deserialize directly into HBM, about 10x faster than re-initializing. This is the closest thing to "boot a warm GPU in 2 seconds." The trade-off is that snapshots are tied to GPU topology, so if Karpenter moves you to a different SKU, you re-checkpoint.

Layer 4: warm pools (min_workers=1)

The simplest mitigation is to keep one replica ready at all times. The cost is one GPU's hourly rate, 24x7. The arithmetic is harsh on small models (you pay $0.85-$1.50/hr to avoid a 30s cold start) and kinder to large ones (you pay $4/hr to avoid a 5-minute cold start). Warm pools usually become mandatory when TTFT P99 has to stay under 60s on a 70B+ model.

Layer 5: tiered loading (ServerlessLLM)

ServerlessLLM treats storage as a hierarchy: NVMe (fast but big), DRAM (medium but tiered), HBM (tiny but instant). Weights are pre-loaded to DRAM and loaded into HBM on demand. The paper reports a 10-200x latency reduction on cold loads versus naive disk-to-HBM. Production adoption is early, though integrations with vLLM exist.

Layer 6: live migration (bonus pattern)

When a node goes away (spot eviction, node drain), the traditional move is to cold-start another replica and drain the request queue. Live migration instead sends the input tokens (kilobytes) to a destination that already has the model loaded, and recomputes the KV cache there. Recomputing is cheaper than shipping gigabytes of KV cache over the network. It applies to disaggregated deployments.

The warm-pool math

For a service with P99 TTFT SLA of 2s, the question is not "warm pool yes or no" but "how many warm replicas, and which paths get them."

High-value interactive paths (live chat, voice agent): min_workers=1-2.
Background batch paths (nightly classification): scale-to-zero accepted, 5-10 minute cold start tolerable.
Premium tier: min_workers per tenant, backed by dedicated capacity.

Measure before optimizing

Cold-start anatomy for a 70B model on a fresh node (illustrative):

Phase	Time	Mitigation
Node provision	50s	Bottlerocket + pre-seeded image, warm pool
Image pull	180s	Pre-seeded data volume (eliminate)
Weights to HBM	75s	Model streamer (halve); GPU snapshot (eliminate)
Engine init	20s	Persistent CUDA graph cache
First forward	3s	Min inherent latency
Total cold	328s
Total with mitigations	~15s	22x reduction

Numbers you should remember

Modal cold start: 2-4s (with GPU snapshots).
Baseten default cold start: 5-10s; sub-second with pre-warming.
Raw 70B cold start: 3-8 minutes.
Run:ai Model Streamer: ~2x weight-load speedup.
ServerlessLLM tiered loading: 10-200x latency reduction (paper numbers).

Check: The Concept0/22

1. What is a pre-seeded image?Define

A term card showing Pre-seeded image, tagged as one layer of cold-start mitigation, with a question asking what it is. — The term as the question names it; its definition is the question.

2. Which AWS-side feature pre-seeds container images so the image-pull step disappears from a cold start?Define

A clue card for an unnamed AWS-side feature: it pre-seeds container images onto new nodes, is referenced from a node's launch configuration, and removes the image-pull step from a cold start, asking which feature this is. — Clues paraphrase the question's own wording; the feature's name is not shown.

3. What does the NVIDIA Run:ai Model Streamer do?Define

A term card showing Run:ai Model Streamer, tagged as an NVIDIA tool that ships native in vLLM 2026, with a question asking what it does. — The term as the question names it; what it does is the question.

4. Which Modal feature gives the closest thing to a warm GPU boot in seconds by deserializing post-load state straight into HBM?Define

A clue card for an unnamed Modal feature: it boots a GPU in a few seconds, restores state from right after the first load, and deserializes that state straight into HBM, asking which feature this is. — Clues paraphrase the question's own wording; the feature's name is not shown.

5. What is a warm pool?Define

A term card showing Warm pool, tagged as one layer of cold-start mitigation, with a question asking what it is. — The term as the question names it; its definition is the question.

6. What does `min_workers=2` do on a serverless platform?Define

A term card showing min_workers set to 2 on a serverless platform, with a question asking what that setting does. — The setting and its value as the question names them; its effect is the question.

7. How does ServerlessLLM's tiered loading cut cold-start latency?Define

The term as the question names it; the mechanism is the question.

8. When a node goes away, what does live migration do instead of cold-starting a replacement and draining the queue?Define

Two crossed-out boxes, cold-start a replacement and drain the request queue, with a note that live migration does something else instead, asking what that is. — The two usual steps are named directly in the question; live migration's own action is not shown.

9. Why does live migration send input tokens to the destination node instead of copying the KV cache?Explain

Two boxes for the same recovery step, one labelled sends input tokens and the other labelled does not send the KV cache, asking why that choice is made. — The choice itself is stated directly in the question; the reason is the question.

10. Why does a GPU snapshot reach a 2-4s cold start on Modal while the Run:ai Model Streamer only roughly halves the weight-load phase?Explain

Two cards, a Modal snapshot at a 2 to 4 second cold start and a Run:ai streamer at roughly 2x weight-load speedup, with a note that one reaches a few seconds total while the other only speeds up one phase, asking why. — Both figures are taken from the lesson's own numbers section; the mechanism that explains the gap is the question.

11. Why must cold-start mitigation be stacked across layers rather than solved with one tool?Explain

A table of the five cold-start phases and their unmitigated seconds, node provision, image pull, weights to HBM, engine init and first forward, asking whether one single tool could fix all five. — Phases and seconds taken directly from the lesson's cold-start anatomy table; no mitigation is shown.

12. What trade-off does a warm pool (`min_workers >= 1`) make?Explain

Both sides taken directly from the lesson's warm-pool layer; no judgment about model size is shown.

13. Why does a Modal GPU snapshot have to be re-checkpointed when Karpenter moves the workload to a different GPU SKU?Explain

A GPU snapshot taken on one SKU, with the workload then moved to a different SKU by Karpenter, asking what has to happen to the snapshot. — The SKU change is taken directly from the question; the consequence is the question.

14. Why is a warm pool a better deal for a 70B model than for a small model?Explain

A table comparing a small model avoiding about 30 seconds of cold start for 0.85 to 1.50 dollars an hour against a 70B model avoiding about 5 minutes for 4 dollars an hour, asking why the deal is better for the 70B row. — Figures taken directly from the lesson's warm-pool layer; the comparison itself is the question.

15. A 70B cold start totals 328 seconds unmitigated and about 15 seconds with the full mitigation stack. Which reduction factor is closest?Apply

Two boxes, an unmitigated cold start of 328 seconds and a fully mitigated one of about 15 seconds, asking for the closest reduction factor. — Both figures taken directly from the lesson's cold-start anatomy table; no ratio is shown.

16. A 70B service must keep P99 TTFT under 60s, and a fresh-node cold start runs about 328s. Which step does the lesson point to?Apply

A 70B service that must keep P99 TTFT under 60 seconds against a fresh-node cold start of about 328 seconds, asking which single step closes the gap. — Both figures taken directly from the question; no candidate step is shown.

17. A warm replica of a 70B model costs about $4/hr. Roughly what does one replica cost over a 30-day month?Apply

A table of three quantities, a 4 dollar hourly rate, 24 hours a day and 30 days in the month, asking for the monthly cost of one such replica. — All three quantities taken directly from the question; no total is shown.

18. A team pre-seeds its node images, yet a 70B cold start still takes over two minutes. Which diagnosis fits?Analyze

A team that pre-seeded its node images, with a measured 70B cold start still over 2 minutes, asking which diagnosis fits. — Both facts taken directly from the question; no breakdown of remaining phases is shown.

19. A disaggregated deployment loses a node to spot eviction, and recovery is slow because the team copies gigabytes of KV cache to the replacement. Which change fits?Analyze

A node evicted by a spot eviction, with the replacement needing the request's state, and the team's current fix shown as copying the gigabytes of KV cache, noted as what makes recovery slow, asking what change fixes it. — The slow current approach is taken directly from the question; the fix is not shown.

20. You run a 70B voice agent with a 2s P99 TTFT SLA, plus a nightly batch classifier on the same model. Traffic is near zero overnight. Which policy fits?Design

A voice agent with a 2 second P99 TTFT SLA and a nightly batch classifier that runs once traffic is near zero, both on the same 70B model, asking which policy fits the two paths. — Both paths taken directly from the question; no min_workers value is shown for either.

21. A premium tier promises a 2s P99 TTFT per tenant on a 70B model, while a batch tier tolerates 5-10 minute starts. Which allocation fits?Design

A premium tier promising 2 second P99 TTFT per tenant and a batch tier tolerating 5 to 10 minute starts, both on the same 70B model, asking which capacity allocation fits both. — Both tiers taken directly from the question; no min_workers value is shown for either.

22. Your 70B service runs on GCP and a 180s image pull dominates its cold start. Which change fits?Design

A GCP cold-start breakdown with image pull at 180 seconds marked as the dominant phase, against the other phases left as some seconds, asking which change targets the dominant phase. — The 180 second image-pull figure is taken directly from the question; no candidate change is shown.

Use It

code/main.py models a cold-start path with and without each mitigation. It reports total cold-start time per stack, then prints a warm-pool cost and a request-rate table for deciding when a warm pool pays for itself.

Check: Use It0/4

1. With Bottlerocket pre-seeding alone, image pull drops to zero and the other 70B phases stay as listed (node 50s, weights 75s, engine 20s, first forward 3s). What does the simulator total?Apply

A table of five phases under Bottlerocket pre-seeding alone: node provision 50 seconds, image pull 0 seconds, weights to HBM 75 seconds unchanged, engine init 20 seconds, first forward 3 seconds, asking for the simulator's total. — Phase values taken directly from the question; no sum row is shown.

2. Add the Run:ai streamer to the pre-seeded stack, which cuts the 75s weight load to 35s. What is the 70B cold start now?Apply

A table of five phases with the Run:ai streamer added to the pre-seeded stack: node provision 50 seconds, image pull still 0 seconds, weights to HBM cut from 75 to 35 seconds, engine init 20 seconds, first forward 3 seconds, asking for the new total. — Phase values taken directly from the question; no sum row is shown.

3. The simulator prints 3.0s for the GPU snapshot stack, while the lesson's table says about 15s with mitigations. A developer concludes one of them is wrong. What explains the gap?Analyze

A simulator printing 3.0 seconds next to a lesson table saying about 15 seconds, with a developer's conclusion that one of them must be wrong, asking what explains the gap instead. — Both figures taken directly from the question; no breakdown of either is shown.

4. A team plans to size its warm pool directly from the break-even table's cold-starts-per-day estimate. What is wrong with that plan?Analyze

The break-even table's req/hr and cold-starts-per-day columns across seven request rates from 1 to 250 per hour, with a plan to size a warm pool directly from the cold-starts-per-day column. — Table captured from a real run of main.py's warm_pool_break_even.

Ship It

This lesson produces outputs/skill-cold-start-planner.md. Give it an SLA, a model size, and a traffic shape, and it picks which mitigations to stack.

Author's Note

I'd run main.py before trusting any number in the table above. It prices a 70B cold start under five stacks and prints each total, so you can see how much each layer buys. Every phase time is a constant in PHASES_70B, not a measurement. The simulator tells you how the layers combine. It cannot tell you what your cluster does.

The two pre-seeding stacks are the ones worth reading closely. Pre-seeding alone drops image pull to zero and leaves the 75s weight load alone. Adding the streamer is what cuts that phase to 35s. The GPU snapshot stack replaces every phase with a small constant, which is why it lands at 3.0s while the table's "~15s with mitigations" stacks several layers instead.

I'd treat the break-even table at the bottom as a sketch. The cold-starts-per-day estimate is a crude function of request rate, so it shows the shape of the argument (a quiet endpoint blows through the drop budget sooner), not a number to size a warm pool from. For that, replace it with your own traffic logs.

Exercises

Run code/main.py. Compute the break-even request rate above which a warm replica is cheaper than paying the cold-start tax via extra request drops at SLO.
You deploy a 13B model with P99 TTFT SLA of 3s. Pick the minimum mitigation stack (fewest layers) that achieves it.
Bottlerocket pre-seeding eliminates image pull but weights still load from snapshot to HBM. Compute wall-clock for a 70B model if the snapshot-backed NVMe reads at 7 GB/s.
Your serverless provider offers GPU snapshots (Modal) and your team refuses because "snapshots leak PII." Argue both sides. What is the realistic risk, and what is the mitigation (ephemeral snapshots, encryption, namespace isolation)?
Design a tiered warm-pool policy: how many warm replicas for paid users, trial users, and batch workloads? Show the math.

Key Terms

Term	What people say	What it actually means
Cold start	"the big pause"	Time from request to first token on a fresh replica
Warm pool	"always-on minimum"	`min_workers >= 1` to keep at least one replica ready
Pre-seeded image	"baked AMI"	Node image with container weights pre-resident
Bottlerocket	"AWS node OS"	AWS container-optimized OS with dual-volume snapshot support
Model streamer	"streaming load"	Overlap weights I/O with compute setup
GPU snapshot	"checkpoint to HBM"	Serialize post-load GPU state; deserialize on restart
Tiered loading	"NVMe + DRAM + HBM"	Hierarchy of storage tiers; load on demand
Live migration	"move tokens"	Transfer input (KB), recompute KV on destination
`min_workers`	"warm replicas"	Serverless minimum keep-alive count
Scale-to-zero	"full serverless"	No cost when idle; accept full cold-start tax