Disaggregated Prefill/Decode — NVIDIA Dynamo and llm-d

You run Llama 3.3 70B on 8 H100s. Under a mixed workload (long prompts, short outputs), GPUs idle during decode because most of the compute went to prefill. Flip the workload to short prompts and long outputs and the opposite happens. Colocating prefill and decode means you over-provision both.

The budget impact is 20-40% of GPU time wasted on the wrong resource. You end up buying H100 compute to run memory-bound decode, or buying H100 HBM bandwidth to run compute-bound prefill. Either way you pay for something you aren't using.

Disaggregation splits prefill and decode onto separate pools, each sized for its own bottleneck. The KV cache moves from the prefill pool to the decode pool over a high-bandwidth interconnect.

Why the bottlenecks differ

Prefill runs the transformer over the full input prompt in one forward pass. Matrix multiplications dominate, so it is compute-bound. H100 FP8 gives ~2000 TFLOPS of useful throughput, and batch efficiency is good because one forward processes many tokens.

Decode generates one token at a time and reads the full weights on each iteration, so it is memory-bandwidth-bound. HBM3 gives ~3 TB/s. Batching only helps at high concurrency, where the weight read amortizes across the batch.

Colocated, you buy GPUs that have to be good at both. An H100 handles both phases, but it costs the same whichever one you're running. At scale you want the prefill pool on H100 (compute-heavy) and the decode pool on H200 (memory-heavy) or on aggressively quantized weights.

The architecture

            ┌──────────────┐
  Request → │    Router    │ ───────────────────────┐
            └──────┬───────┘                        │
                   │                                │
                   ▼ (prompt only)                  │
            ┌──────────────┐    KV cache    ┌───────▼──────┐
            │ Prefill pool │ ─── NIXL ────► │ Decode pool  │
            │  (compute)   │                │  (memory)    │
            └──────────────┘                └──────┬───────┘
                                                   │ tokens
                                                   ▼
                                                 Client

NIXL is NVIDIA's inter-node transport. It uses RDMA/InfiniBand when available and falls back to TCP otherwise. The transfer latency is real: typically 20-80 ms for the KV cache of a 4K-token prompt on 70B FP8. That is why short prompts don't justify disaggregation. The transfer tax exceeds the savings.

Dynamo vs llm-d

NVIDIA Dynamo (GTC 2025 announce, 1.0 GA):

• Sits above vLLM, SGLang, and TRT-LLM as an orchestrator.
• The Planner Profiler measures the workload and the SLA Planner auto-configures prefill:decode ratios.
• Rust core, Python extensibility.
• Throughput gains: NVIDIA reports 6x for DeepSeek-R1 MoE on GB200 NVL72 + Dynamo in the medium-latency regime (developer.nvidia.com, 2025-06). Community reports of "up to 30x" on full Blackwell + Dynamo + DeepSeek-R1 stacks lack a single primary source, so read them as directional.
• GB300 NVL72 + Dynamo: up to 50x MoE throughput vs Hopper per the Dynamo product page (developer.nvidia.com, undated).

llm-d (Red Hat + AWS, Kubernetes-native):

• Prefill, decode, and router run as independent Kubernetes Services.
• Per-role HPA scales on queue depth (prefill) and KV utilization (decode).
• topologyConstraint packDomain: rack packs prefill+decode cliques onto the same rack for high-bandwidth KV transfer.
• llm-d 0.5 (2026) adds hierarchical KV offloading, cache-aware LoRA routing, UCCL networking, and scale-to-zero.

Pick Dynamo if you want a managed orchestrator that sits above your engines. Pick llm-d if you want Kubernetes-native primitives and are committed to the CNCF ecosystem.

Economics

This is an internal composite, not a single published case study, so treat it as an order-of-magnitude anchor:

• $2M/year inference spend on colocated serving.
• Switched to disaggregated with Dynamo.
• Same request volume, same P99 latency SLA.
• Reported savings: $600K–$800K/year (30–40% reduction).
• No new hardware.

I synthesized this figure from multiple customer disclosures. The closest published data points are Baseten's 2x faster TTFT and 61% higher throughput with Dynamo KV routing (baseten.co, 2025-10), and VAST + CoreWeave's projection of 60–130% more tokens/$ at a 40–60% KV hit rate (vastdata.com, 2025-12). The savings come from right-sizing each pool, so prefill-heavy workloads (RAG with 8K+ prefixes) benefit more than balanced ones.

When NOT to disaggregate

• Prompts under 512 tokens and outputs under 200 tokens: the transfer tax dominates the gain.
• Small cluster (under 4 GPUs): not enough pool diversity.
• A team that cannot operate two GPU pools with per-role scaling. Dynamo helps, but not trivially.
• No RDMA fabric: the TCP transfer tax is heavier.

The router integrates with Phase 17 · 11

Disaggregated routers are KV-cache-aware (Phase 17 · 11). A request lands on the decode pool that holds its prefix, and if nothing matches it flows prefill to decode. Hit rate and disaggregation compound, because the cache-aware router decides whether a new prefill is needed at all.

MoE on Blackwell is where the big numbers are

GB300 NVL72 + Dynamo shows 50x MoE throughput over Hopper baselines. MoE expert routing is compute-heavy on prefill but memory-heavy on decode (expert caches), so disaggregation is a double win. Frontier model serving in 2026 is MoE-dominant (DeepSeek-V3, future GPT-5 variants).

Numbers you should remember

Benchmark numbers drift. NVIDIA and the inference stack vendors post updated results every quarter, so re-check before quoting.

• DeepSeek-R1 on GB200 NVL72 + Dynamo: ~6x throughput vs baseline in the medium-latency regime (developer.nvidia.com, 2025-06). Community "up to 30x" claims on full Blackwell + Dynamo stacks are directional aggregates without a single primary source.
• GB300 NVL72 + Dynamo: up to 50x MoE throughput vs Hopper (developer.nvidia.com, undated).
• Savings anchor (internal composite, not a single case study): $600-800K/year off a $2M annual spend at constant SLA.
• Disaggregation threshold: prompts >512 tokens + outputs >200 tokens.
• KV transfer via NIXL: 20-80 ms for 4K-prompt KV on 70B FP8.

main.py simulates colocated vs disaggregated serving for a single request. It prints per-request latency for colocated, disaggregated over RDMA, and disaggregated over TCP, plus which side wins, across a sweep of prompt lengths.

This lesson produces outputs/skill-disaggregation-decider.md. Given a workload and a cluster, it decides whether to disaggregate.

1. Run main.py. At what prompt length does disaggregation beat colocation? The shipped cases never flip, so change the constants or the sweep until they do.
2. Design the prefill pool and decode pool for a RAG service with P99 prefix length 8K, output 300.
3. Dynamo vs llm-d: pick one for a pure-Kubernetes shop with no Python runtime preference.
4. Compute KV transfer cost: 4K prefill on 70B FP8 = ~500 MB KV. At RDMA 100 GB/s, transfer = 5 ms. At TCP 10 GB/s = 50 ms. Which matters for your SLA?
5. MoE expert routing changes KV access patterns. How does disaggregation behave with MoE that activates different experts per token?

• NVIDIA — Introducing Dynamo
• NVIDIA — Disaggregated LLM Inference on Kubernetes
• TensorRT-LLM Disaggregated Serving blog
• llm-d GitHub
• llm-d 0.5 release notes