TAI BUI
Back to curriculum
LearnPython

vLLM Production Stack with LMCache KV Offloading

vLLM's production-stack is the reference Kubernetes deployment: router, engines, and observability wired together. LMCache is the KV-offloading layer that pulls KV cache out of GPU memory and reuses it across queries and engines, first in CPU DRAM, then on disk or Ceph. The vLLM 0.11.0 KV Offloading Connector (January 2026) makes that offload asynchronous and pluggable through the Connector API (v0.9.0+), so offload latency stays out of the user's path. LMCache can pay off even without shared prefixes: when a GPU runs out of KV slots, a preempted request can be restored from CPU instead of re-running prefill. In the published benchmarks on 16x H100 (80GB HBM) across 4 a3-highgpu-4g, native CPU offload and LMCache both improve throughput substantially once the KV cache exceeds HBM. At a low KV footprint the configs match baseline, with a small overhead.

TIME: ~60 minutesPREREQUISITES: Phase 17 · 04 (vLLM Serving Internals), Phase 17 · 06 (SGLang/RadixAttention)

Learning Objectives

  • Diagram the vLLM production-stack layers: router, engines, KV offload, observability.
  • Explain the KV Offloading Connector API (v0.9.0+) and how the 0.11.0 asynchronous path hides offload latency.
  • Quantify when LMCache CPU-DRAM helps (KV > HBM) vs adds overhead (KV small enough to fit HBM).
  • Pick between native vLLM CPU offload and LMCache connector given deployment constraints.

The Problem

Your vLLM serving shows GPUs at 100% HBM, with preemption events whenever concurrency climbs. Requests get evicted and requeued, and you re-prefill the same 2K-token prompt four times in a minute. GPU compute goes to redundant prefills, so goodput sits well below raw throughput.

Adding GPUs costs linearly, and you can't add HBM to a GPU you already own. CPU DRAM is a different story. One socket can hold 512 GB or more, and its latency is orders of magnitude worse than HBM, but that is fine for KV cache that only needs to stay temporarily warm.

LMCache extracts KV cache to CPU DRAM so preempted requests recover quickly. It also lets engines share cache for repeated prefixes, so each engine doesn't re-prefill them on its own.

Warm-up0/2
1. Compared with a GPU's HBM, what is the usual trade-off of spilling data into a server's CPU DRAM?Warm-up
A GPU memory box and a server's CPU-side memory box, drawn at the same size, with an arrow from the GPU box to the CPU box labeled overflow. Two lines below ask how capacity and access speed compare, both left open.
Setup: the two parts, drawn the same size on purpose. Neither comparison is shown.
2. Two prompts begin with the same 1,500-token system prompt and then diverge. With prefix caching, what can the second request reuse?Warm-up
Two rows, each a shared 1,500-token system prompt box followed by its own different continuation box. A line asks what request B can reuse from request A's work on the shared span.
Setup: the two requests and their shared leading span. What gets reused is left open.
Check: The Problem0/1
1. In this lesson's setting, what is preemption?Define
An HBM gauge shown full, a box for request R already running with cached KV, an arrow down to a back-of-queue box, and a dashed curved arrow looping back toward R's original position labeled prefilled again with a question mark. A line asks what caused R to end up back at the start.
Setup: one request's round trip while HBM sits full. What causes it is left open.

The Concept

vLLM production-stack

github.com/vllm-project/production-stack is the reference Kubernetes deployment. It ships as a Helm chart plus an operator, and it has five parts:

  • Router. Cache-aware (Phase 17 · 11), and it consumes KV events.
  • Engines. vLLM workers, one per GPU or per TP/PP group.
  • KV cache offload. An LMCache deployment or the native connector.
  • Observability. Prometheus scrape, Grafana dashboards, OTel traces.
  • Control plane. Service discovery, config, rolling updates.

The KV Offloading Connector API (v0.9.0+)

vLLM 0.9.0 introduced a Connector API for pluggable KV cache backends. The engine offloads blocks to the connector, and the connector stores them (RAM, disk, object storage, LMCache). When a request needs a block, the connector loads it back.

vLLM 0.11.0 (January 2026) adds an asynchronous offload path, so offload can happen in the background and the engine usually doesn't block on it. End-to-end latency and throughput still depend on workload shape, KV cache hit rate, and system pressure. vLLM's own notes say custom-kernel offload can degrade throughput at low hit rates, and that async scheduling has known interaction issues with speculative decoding.

Native CPU offload vs LMCache

Native vLLM CPU offload is engine-local. It stores KV blocks in host RAM, which is quick to set up and has no network hop. It doesn't cross engines.

LMCache connector works at cluster scale. It stores blocks in a shared LMCache server (CPU DRAM plus a Ceph/S3 tier), and any engine can read them. The 16x H100 benchmarks are published.

Pick native when a single engine has HBM pressure. Pick LMCache when multiple engines share prefixes, as with RAG over common system prompts or multi-tenant serving with shared templates.

Benchmark behavior

The test spreads 16x H100 (80 GB HBM) across 4 a3-highgpu-4g:

  • Low KV footprint (short prompts, low concurrency): the configs match baseline, and LMCache adds roughly 3-5% overhead.
  • Moderate footprint: LMCache starts to help on prefix reuse across engines.
  • KV exceeds HBM: native CPU offload and LMCache both improve throughput substantially. LMCache gains more because of cross-engine sharing.

When LMCache is decisive

  • Multi-tenant serving where tenants share system prompts.
  • RAG where document chunks repeat across queries.
  • Fine-tuned variants (LoRA) on one base model, where reusing the base model's KV cuts redundant work.
  • Preemption-heavy workloads, where restoring from CPU is cheaper than re-prefilling.

When not to enable it

  • Small HBM pressure. You pay the overhead and get no benefit.
  • Short contexts (<1K tokens), where transfer time exceeds re-prefill time.
  • A single-tenant, single-prompt workload, which has no reuse to capture.

Integration with disaggregated serving

Phase 17 · 17 disaggregated serving compounds with LMCache. KV transferred from the prefill pool to the decode pool lands in LMCache if nothing uses it, and later queries pull it from there. The Phase 17 · 11 cache-aware router can then route to the engine whose local or LMCache-shared cache matches.

Numbers to remember

  • vLLM 0.9.0: Connector API shipped.
  • vLLM 0.11.0 (Jan 2026): asynchronous offload path. The latency impact depends on workload, KV hit rate, and system pressure, so treat it as a tendency rather than a guarantee.
  • 16x H100 benchmark: LMCache helps when the KV footprint exceeds HBM.
  • Small HBM pressure: 3-5% overhead without benefit.
Check: The Concept0/21
1. What are the layers of the vLLM production-stack reference deployment?Define
A Kubernetes cluster boundary containing five unlabeled component boxes, named only part A through part E, in a non-sequential layout. A line asks which five named parts this lesson gives and in what roles.
Setup: the cluster boundary and five slots. None of the five names appear.
2. What vLLM API introduced pluggable KV cache backends?Define
An engine box pointing to an unlabeled interface box, which has dashed arrows down to three backend boxes: host RAM, disk or object storage, and LMCache. A line asks for the interface's name.
Setup: the engine and its three possible backends. The interface's name is left open.
3. What is native vLLM CPU offload?Define
A dashed boundary labeled one engine, containing a GPU box out of HBM room with an arrow down to a host RAM box on the same machine. Nothing else appears outside the boundary. A line asks what this pattern is called.
Setup: one engine's own store-and-reload path. No other engine is drawn.
4. What is LMCache in the vLLM production stack?Define
Four engine boxes in a row, each with a dashed arrow down to one unlabeled shared box beneath them. A line asks what the shared store is called and who can read it.
Setup: four engines and one shared target. Its name and who may reach it are left open.
5. What does the vLLM 0.11.0 (January 2026) release add to the KV offload path?Define
Two tracks for one engine: a row of decode steps and, below, a bar labeled offload copy with a question mark next to it. A line asks whether offload runs alongside decode or decode waits for it.
Setup: the two tracks happening around the same time. Their timing relationship is left open.
6. What is prefix reuse?Define
Three request boxes, each with a shared-colored leading segment and a different trailing segment, all with dashed arrows converging on one unlabeled box below. A line asks what this pattern is called.
Setup: three requests sharing a beginning. The name for the pattern is left open.
7. What is the Ceph tier in an LMCache deployment?Define
Three stacked bars: GPU HBM on top labeled fastest and smallest, CPU DRAM in the middle labeled bigger and slower, and an unlabeled bottom bar. A line asks what sits in the bottom tier and what kind of storage it is.
Setup: the hierarchy's top two tiers, named. The bottom tier is left open.
8. Why does the production-stack router consume KV events and act cache-aware?Explain
A router box above three engine boxes, each holding unspecified cached blocks, with one incoming request arrow pointing at the router. A line asks the reason the router consumes KV events.
Setup: a router and three engines, none of their cached contents spelled out. The reason is left open.
9. Why does the 0.11.0 asynchronous path usually avoid stalling the engine during offload?Explain
Two tracks for one engine, a row of decode steps and a dashed-outlined offload-copy bar with a label asking whether it sits on the critical path. A line asks why the engine usually doesn't stall here.
Setup: decode and offload happening around the same time. Whether offload sits on the critical path is left open.
10. When should you pick LMCache over native CPU offload?Explain
Two scenario cards: one engine with HBM pressure and no shared prefixes, and several engines sharing the same prefixes across tenants or RAG chunks. A line asks which one favors LMCache over native offload.
Setup: the two scenarios. Which favors which backend is left open.
11. How does LMCache help when serving several LoRA fine-tuned variants built on the same base model?Explain
A base model box with three LoRA variant boxes branching from it, each labeled own adapter on top, connected by dashed arrows. A line asks how LMCache helps serving all of them.
Setup: one base model and its variants. How LMCache helps is left open.
12. What happens to LMCache's benefit when the KV footprint stays well below HBM?Explain
A box stating the KV footprint stays well below HBM, next to a box noting LMCache is enabled anyway. A line asks what happens to LMCache's benefit in this case.
Setup: the low-footprint scenario, exactly as given. No figure is shown.
13. Why does the lesson advise against LMCache for contexts under about 1K tokens?Explain
Two boxes, time to move the KV and time to re-run the short prefill, both marked unknown. A line asks which one wins for a context under about 1K tokens.
Setup: the two competing durations for a short context. Neither is resolved.
14. Why is native CPU offload the better pick for a single engine under HBM pressure?Explain
A single highlighted engine box under HBM pressure, with no other engines drawn. A line asks whether native offload or LMCache fits here, and why.
Setup: one engine, alone. Which backend fits is left open.
15. Why does LMCache compose with disaggregated serving (Phase 17 · 17)?Explain
A prefill pool box with an arrow to a decode pool box, plus a dashed side box asking where leftover KV ends up if the decode pool doesn't use it. A line asks who can use it later.
Setup: the prefill-to-decode transfer. What happens to unused KV is left open.
16. A team enables LMCache on a single-tenant service with one prompt shape, short prompts, and HBM well under its limit. Throughput drops a few percent. What explains it?Analyze
A single-tenant service with one prompt shape, short prompts and HBM well under its limit, with LMCache enabled anyway. A note says throughput drops a few percent. A line asks what explains it.
Setup: the scenario, exactly as given. No specific overhead figure or mechanism is shown.
17. A team enables native CPU offload on 4 engines and expects engine B to stop re-prefilling a system prompt that engine A already computed. B still re-prefills it. What was the wrong assumption?Analyze
Engine A, which computed and offloaded a system prompt, next to engine B, which expects to reuse it but re-prefills anyway. A line asks what was assumed wrong.
Setup: the surprising outcome, exactly as given. No reason is shown.
18. A teammate says the 0.11.0 async path makes offload free. Which observation from vLLM's notes argues against that?Analyze
A speech bubble claiming the async path makes offload free, next to a box marked vLLM's own notes with a question mark. A line asks which observation argues against the claim.
Setup: the claim, exactly as stated in the question. No counter-evidence is shown.
19. A team runs 4 vLLM engines for a RAG product. Every query starts with the same 3K-token system prompt, GPUs sit at 100% HBM, and preemptions keep re-prefilling that prompt. What is the best first move?Design
Four engines in a RAG product where every query starts with the same 3K-token system prompt, GPUs sit at 100 percent HBM, and preemptions keep re-prefilling that prompt. A line asks for the best first move.
Setup: the scenario, exactly as given. No move is shown.
20. One engine serves long single-user chats whose KV grows past HBM, with no shared prefixes and no other engines. Which setup fits?Design
A single highlighted engine serving long single-user chats whose KV grows past HBM, with a note that there are no shared prefixes and no other engines. A line asks which setup fits.
Setup: the scenario, exactly as given. No setup choice is shown.
21. A new service has prompts around 600 tokens, one tenant, and HBM at 40 percent. A colleague wants LMCache on just in case. What is the best call?Design
A new service with prompts around 600 tokens, one tenant and HBM at 40 percent, next to a speech bubble from a colleague saying enable LMCache just in case. A line asks for the best call.
Setup: the scenario and the colleague's suggestion, exactly as given. No call is shown.

Use It

code/main.py simulates 200 requests over 4 engines that draw from 6 shared prefixes, under three configs: no offload, engine-local CPU offload, and a shared LMCache. It reports total time, prefill time, re-prefills avoided, and speedup against the no-offload baseline. It doesn't model HBM utilization (the per-engine capacity constant is declared but not used), so it can't give you a break-even point. Treat it as a model of reuse across engines, not of HBM pressure.

Check: Use It0/5
1. In main.py, a 2,000-token request misses its engine's local set but hits the shared LMCache (16-token blocks, 3.0 ms per block). What prefill_ms is charged?Apply
The real LMCACHE branch line from main.py, with the real KV_BLOCK_TOKENS and LMCACHE_TIME_MS_PER_BLOCK constants given, and a 2000-token request that misses locally but hits the shared cache. The charged time is left as a question mark.
Setup: the real branch and constants, code copied from main.py. The charged time is left open.
2. For that same 2,000-token prompt, how does a fresh prefill compare with the LMCache hit in main.py?Apply
Two boxes for the same 2000-token prompt, fresh prefill and LMCache hit, both with an unknown millisecond cost. A line asks which is larger and by about how much.
Setup: the two paths for one prompt. Both costs are left open.
3. In the LMCACHE config of main.py, a request lands on an engine whose local set already holds its prefix_id. What happens?Apply
The real LMCACHE branch order from main.py: a local-cache check comes first, with the shared-cache check only reached in an elif below it. A line asks what happens to a request that hits the first branch.
Setup: the real branch order, code copied from main.py. What the first branch does is left open.
4. In the CPU_OFFLOAD config, a 4,000-token request misses locally. Beyond its 100 ms prefill, how much extra offload time does main.py add?Apply
The real CPU_OFFLOAD miss branch from main.py, with a 4000-token request that misses locally and a 100 ms base prefill given. The extra offload time is left as a question mark.
Setup: the real branch, code copied from main.py. The extra time beyond the 100 ms base is left open.
5. In the simulator run, LMCache avoids the most re-prefills yet finishes at 0.98x to 0.99x of baseline. Which diagnosis fits?Analyze
Two cards about LMCache in the same run: it avoids the most re-prefills, and it finishes at 0.98x to 0.99x of baseline. A line asks which diagnosis fits.
Setup: the two facts, exactly as given in the question. No cause is shown.

Ship It

This lesson produces outputs/skill-vllm-stack-decider.md. Given a workload shape and a vLLM deployment, it decides between native offload, LMCache, and neither.

Author's Note

I'd run main.py first and read the output skeptically. LMCache avoids the most re-prefills (194, against 176 for CPU offload and roughly 120 with no offload), yet it finishes slightly behind the baseline at 0.98x to 0.99x. The cause is the cost model. A shared-cache hit is charged 3.0 ms per 16-token block, which works out to about 5 tokens/ms, while a fresh prefill runs at 40 tokens/ms. Every LMCache hit is therefore slower than the prefill it replaces, and prefill time actually rises (14800 ms against roughly 9000 ms with no offload). Decode time also dominates the total in the toy, so the differences stay small. The engine-local path adds only 0.15 ms per block on a miss, so CPU offload comes out slightly ahead at about 1.02x. The numbers say that avoiding a prefill is not the same as saving time once you pay to move the blocks.

That's the trade I'd keep in mind from the body. The 3-5% overhead at low KV footprint and the preference for native offload on a single engine are the same point at cluster scale, and the toy shows a small version of it.

I'd also keep the simulator in proportion. Only the no-offload config evicts anything (it keeps at most 4 prefixes per engine, and which one it drops varies from run to run, so its numbers wobble slightly), and the HBM capacity constant never enters the arithmetic. It can't tell you when LMCache starts paying under memory pressure. For that I'd measure the real stack with my own prompts, since hit rate and prompt length decide the result more than anything in this file.

Exercises

  1. Run code/main.py. Which config avoids the most re-prefills, and does it also finish first? (The no-offload row shifts a little between runs, so compare against the baseline rather than exact counts.) Then change hbm_capacity_blocks_per_engine and check whether anything moves. What does that tell you about what the simulator models?
  2. A tenant shares a 6K-token system prompt across 200 queries/hour. Compute expected LMCache savings per tenant.
  3. The LMCache server is a single point of failure. Design the HA strategy (replicas, fallback to native).
  4. LMCache stores to Ceph on spinning disk. For a 4K-token KV at 70B FP8 (500 MB), what's the read time vs re-prefill?
  5. Argue whether the vLLM 0.11.0 asynchronous path is "free". Where does the overhead hide?

Key Terms

TermWhat people sayWhat it actually means
Production-stack"the reference deployment"vLLM's Kubernetes Helm chart + operator
Connector API"KV backend interface"vLLM 0.9.0+ pluggable KV store interface
Native CPU offload"engine-local spill"Store KV in host RAM of same engine
LMCache"cluster KV cache"Cross-engine KV cache server on CPU DRAM + disk
0.11.0 async"non-blocking offload"Offload hidden behind engine stream
Preemption"evict to make room"KV cache shuffle when HBM full
Prefix reuse"same system prompt"Multiple queries share beginning; cache hit
Ceph tier"disk tier"Durable storage below DRAM in the cache hierarchy

Further Reading

Code

Ships With