vLLM Serving Internals: PagedAttention, Continuous Batching, Chunked Prefill

A naive PyTorch serve loop runs one request at a time: tokenize, prefill, decode until EOS, return. At one user this works. At one hundred, it is a queue of patient people. The obvious fix is static batching, which pads every request to the longest prompt in the window, pads every decode to the longest expected output, and stalls the whole batch on the slowest sequence. You pay for padding you never use, and fast requests wait for slow ones.

vLLM attacks three problems at once. PagedAttention stops KV cache fragmentation from eating 60-80% of GPU memory the way classic contiguous allocation does. Continuous batching lets requests join and leave the batch between decode iterations, so the batch is full of real work. Chunked prefill breaks a 32k-token prompt into ~512-token slices that interleave with decode, so one long prompt does not freeze every other sequence's next token.

The 2026 production default has all three on. Learn what each one does, because the failure modes live in the scheduler, not the model.

PagedAttention as a virtual memory system

A KV cache is num_layers × 2 × num_heads × head_dim × seq_len × bytes_per_element per sequence. For Llama 3.3 70B at 8192 tokens, that is roughly 1.25 GB per sequence in BF16. If you pre-reserve 8192 slots for every request but the average request only uses 1500 tokens, you waste roughly 82% of the HBM you reserved. Classic batching pays this waste.

PagedAttention borrows the idea from OS virtual memory. KV cache is not contiguous per sequence. It is allocated in fixed-size blocks (default 16 tokens). Each sequence has a block table that maps its logical token positions to physical block IDs. When a sequence grows past its allocated blocks, one more block is added. When it finishes, its blocks return to the pool.

Fragmentation drops from 60-80% (classic) to under 4% (PagedAttention). There is no flag to enable PagedAttention. It is the only allocator vLLM ships. The knob is --gpu-memory-utilization (default 0.9), which tells vLLM how much HBM to reserve for KV blocks after loading weights and activations.

Continuous batching at the iteration level

The old "dynamic batching" waited for a window (say 10 ms) to fill a batch, then ran prefill + decode + decode + decode until every sequence finished. Fast sequences left early and sat idle while the GPU finished the slow ones.

Continuous batching operates between each decode step. Call the set of running sequences the RUNNING list. At each iteration:

1. Any sequence in RUNNING that just hit EOS or max_tokens is removed.
2. The scheduler looks at the waiting queue. If there are free KV blocks, it admits new sequences (prefill or resumed).
3. The forward pass runs on whatever is now in RUNNING, emitting one new token per sequence.

The batch is not padded to a fixed size. Sequences at different positions in their output share one fused forward. In 2026 vLLM this is called the V1 scheduler. The key invariant: the scheduler runs once per decode iteration, not once per request.

Chunked prefill protects the ITL tail

Prefill is compute-bound. A 32k-token prompt on Llama 3.3 70B takes ~800 ms of pure prefill on one H100. While prefill runs, decode tokens for every other sequence in the batch wait. In a serving loop, the first-token latency (TTFT) of one long prompt becomes the inter-token latency (ITL) blip for dozens of other users.

Chunked prefill splits prefill into fixed-size chunks (default 512 tokens) and schedules each chunk as a unit. Between chunks the scheduler can advance decode sequences by one token. You trade a small absolute prefill latency hit (a few ms per chunk) for much lower decode-time jitter. P99 ITL under mixed load drops from ~50 ms to ~15 ms in published benchmarks.

The three defaults interact

All three features assume each other. PagedAttention gives the scheduler a fine-grained KV resource to trade against. Continuous batching needs that fine-grained resource so admitting a new sequence does not force a global reshuffle. Chunked prefill is a decision the scheduler makes on the same RUNNING list. It is one more scheduler policy, not a separate system.

You do not need to know every flag. You need to know what the scheduler optimizes: goodput under KV-block budget, subject to chunked prefill slicing.

The 2026 v0.18.0 gotcha

In vLLM v0.18.0 you cannot combine --enable-chunked-prefill with draft-model speculative decoding (--speculative-model). The documented exception is N-gram GPU speculative decoding in the V1 scheduler. Teams that flip every flag on without reading the release notes get a run-time error at startup, not a soft regression. If your speculative gain was worth enabling chunked prefill for, revisit the choice. The better answer in 2026 is often EAGLE-3 without chunked prefill, not a draft model plus chunked prefill that fails at startup.

Numbers you should remember

• Llama 3.3 70B FP8, H100 SXM5, 128 concurrent, all three on: 2,200-2,400 tok/s.
• Same model, default vLLM (no chunked prefill): ~1,800 tok/s.
• Same model, naive PyTorch forward loop: ~600 tok/s.
• KV fragmentation waste under PagedAttention at production load: <4%.
• P99 ITL under mixed load: ~15 ms with chunked prefill, ~50 ms without.

What the scheduler looks like

while True:
    finished = [s for s in RUNNING if s.is_done()]
    for s in finished: release_blocks(s); RUNNING.remove(s)

    while WAITING and have_free_blocks_for(WAITING[0]):
        s = WAITING.pop(0)
        allocate_initial_blocks(s)
        RUNNING.append(s)

    # schedule prefill chunks + decode in one batch
    batch = []
    for s in RUNNING:
        if s.in_prefill:
            batch.append(next_prefill_chunk(s))   # e.g. 512 tokens
        else:
            batch.append(decode_one_token(s))     # 1 token

    run_forward(batch)                            # one fused GPU call

main.py follows this loop in stdlib Python with fake token counts and fake forward latency. Running it shows how chunked prefill keeps decode sequences moving during a long prefill.

main.py simulates a vLLM-style scheduler in four modes on one 60-request workload:

• NAIVE: one request at a time, no batching.
• STATIC: pad and wait, classic batching with a batch of 16.
• CONTINUOUS: iteration-level admission and release.
• CONTINUOUS + CHUNKED: prefill slices interleaved with decode.

Each row prints throughput (tokens per virtual second), mean TTFT, and P99 ITL. The latency constants are illustrative, not measured, so read the shape of the rows and not the absolute values. The cleanest signal is P99 ITL between the two continuous rows, where chunking pulls the tail down. Throughput is murkier: the toy's STATIC mode charges decode cost on a simplified curve, so check the output before you trust any claim that one row wins everywhere.

This lesson produces outputs/skill-vllm-scheduler-reader.md. Given a serving config (batch size, KV memory utilization, chunked prefill size, speculative config), it produces a scheduler diagnosis that names which of the three defaults is the bottleneck and what to tune.

1. Run main.py. Compare STATIC to CONTINUOUS on a workload with mixed short and long requests. Where does the gap come from: prefill efficiency, decode efficiency, or tail latency?
2. Modify the toy scheduler to add --max-num-batched-tokens. What is the right value for an H100 running Llama 3.3 70B FP8? (Hint: it is a function of KV block size and number of free blocks, not raw HBM.)
3. Re-read the vLLM v0.18.0 release notes. Which combinations of flags are mutually exclusive? List them.
4. Compute the KV cache fragmentation waste for a trace of 1,000 requests with mean 1,500 output tokens, std 600 tokens, under (a) contiguous per-request allocation at 8192 max, (b) PagedAttention with 16-token blocks.
5. Explain in one paragraph why chunked prefill helps P99 ITL but not throughput in isolation. Where does the throughput win come from in practice?

• vLLM documentation: Speculative Decoding: official source on chunked-prefill and speculative-decoding compatibility.
• vLLM Release Notes (NVIDIA): 2026 release cadence and version-specific behavior.
• vLLM Blog: PagedAttention: the original write-up that still defines how to think about the allocator.
• PagedAttention paper (arXiv:2309.06180): fragmentation analysis and scheduler design.
• Aleksa Gordic: Inside vLLM: detailed V1 scheduler walkthrough with flame graphs.