TAI BUI
Back to curriculum
CapstonePython (serving), C++ / CUDA (kernel inspection), YAML (configs)

Capstone 14 — Speculative-Decoding Inference Server

EAGLE-3 in vLLM 0.7 ships 2.5-3x throughput on real traffic. P-EAGLE (AWS 2026) pushes parallel speculation further, SGLang's SpecForge trains draft heads at scale, Red Hat's Speculators hub publishes aligned drafts for common open models, and TensorRT-LLM treats speculative decoding as first-class on NVIDIA. The 2026 production stack is vLLM or SGLang with EAGLE-family drafts, FP8 or INT4 quantization, and HPA on queue-wait. The capstone: serve two open models at 2.5x+ baseline throughput and publish a full tail-latency report.

TIME: 30 hoursPREREQUISITES: Phase 3 (deep learning), Phase 7 (transformers), Phase 10 (LLMs from scratch), Phase 17 (infrastructure)

Learning Objectives

  • Serve Llama 3.3 70B in FP8 under vLLM 0.7 with an EAGLE-3 draft head and measure the speedup against a non-speculative baseline run on the same benchmark.
  • Report acceptance rate per traffic distribution (ShareGPT, HumanEval, domain data) and explain why it drifts when the traffic mix changes.
  • Publish p50 and p99 latency at batch 1, 8, and 32 with and without speculation, showing the tail cost of rejected drafts.
  • Deploy the server on Kubernetes with HPA on queue-wait, and compute cost per 1M tokens against hosted APIs.

Problem

Speculative decoding is a serving technique where a cheap draft model proposes several tokens ahead and the target model checks all of them in a single forward pass, keeping the prefix that matches. By 2026 it is a commodity. EAGLE-3 draft heads train on the target model's hidden states and predict N tokens ahead. Acceptance rates of 60-80% translate to 2-3x end-to-end throughput. vLLM 0.7 integrates it natively, SGLang + SpecForge gives you the training pipeline, and Red Hat's Speculators publishes aligned drafts for Llama 3.3 70B, Qwen3-Coder-30B MoE, and GPT-OSS-120B.

The model is the easy part. The craft is in the serving operations. Acceptance rate drifts with the traffic distribution (ShareGPT vs code vs domain data). Tail latency under rejection is worse than without speculation, so steady-state tokens/sec is not enough: you have to report p99 at multiple batch sizes. And cost per 1M tokens against the Anthropic and OpenAI APIs is what makes the whole exercise credible.

Warm-up0/3
1. Why is generating text one token at a time slow on a GPU, even for a large model?Warm-up
2. What is the main effect of quantizing a model's weights from 16-bit to 8-bit?Warm-up
3. Which Kubernetes object adds or removes pod replicas automatically based on a metric?Warm-up
Check: Problem0/7
1. What does EAGLE-3 do that ngram drafts do not?Define
2. What does the Speculators hub provide?Define
3. Why does acceptance rate move when the traffic mix shifts from chat to code?Explain
4. Why does the deliverable include $/1M tokens against hosted APIs?Explain
5. Why is p99 tail latency reported across batch sizes 1, 8, and 32?Explain
6. Your server hit 2.8x on ShareGPT, but after most traffic became code completion, acceptance fell and p99 rose. What is the likeliest cause and first response?Analyze
7. You must serve a code-heavy workload where EAGLE-3 acceptance is volatile and the p99 target is strict. Which design fits best?Design

Concept

Speculative decoding has two layers. A draft model (an EAGLE-3 head, ngram lookup, or a smaller target-aligned model) proposes k candidate tokens per step. The target model verifies all k in one pass, and any accepted prefix replaces the greedy path. Acceptance rate depends on how well the draft is aligned with the target and on the input distribution.

EAGLE-3 beats ngram drafts on most traffic. P-EAGLE adds parallel speculation, which gives you deeper draft trees. The cost is tail latency: when a draft is rejected, the verify pass was larger than it would have been without speculation, so P99 gets worse. That is why the serving config has to report latency bucketed by batch size.

Deployment is Kubernetes. vLLM 0.7 runs one replica per GPU or per tensor-parallel shard, and HPA autoscales on queue-wait rather than CPU. FP8 (Marlin) and INT4 (AWQ) quantization keep GPU memory inside an H100 / H200 envelope. The end-to-end report covers throughput, acceptance rate, p50/p99 at batch 1/8/32, and $/1M tokens.

Check: Concept0/9
1. What is a draft model in speculative decoding?Define
2. What does acceptance rate measure?Define
3. What does P-EAGLE add over serial EAGLE-3?Define
4. What does the queue-wait metric report?Define
5. Why can speculation make p99 worse than running without it?Explain
6. Load triples. Requests pile up in the queue while CPU stays low. Which signal should trigger scale-out?Analyze
7. A team ships after measuring 2.9x tokens/sec at batch 32. Users later report occasional very slow responses. Which measurement was most likely missing?Analyze
8. Traffic is unchanged, but acceptance drops after the target is upgraded to a newer version while the original draft head stays. What is the most direct explanation?Analyze
9. A stakeholder wants proof that your speculative server beats a hosted API. Which report is the most convincing?Design

Architecture

request ingress
    |
    v
vLLM server (0.7) or SGLang (0.4)
    |
    +-- draft: EAGLE-3 heads | P-EAGLE parallel | ngram fallback
    +-- target: Llama 3.3 70B | Qwen3-Coder-30B | GPT-OSS-120B
    |     quantized FP8-Marlin or INT4-AWQ
    |
    v
verify pass: batch k draft tokens through target
    |
    v (accept prefix; resample for rejected suffix)
    v
token stream back to client
    |
    v
Prometheus metrics: throughput, acceptance rate, queue wait, latency p50/p99
    |
    v
HPA on queue-wait metric
Check: Architecture0/1
1. In the architecture, what happens to the draft tokens after the target's verify pass?Define

Stack

  • Serving: vLLM 0.7 or SGLang 0.4
  • Speculative methods: EAGLE-3 draft heads, P-EAGLE parallel speculation, ngram fallback
  • Draft training: SpecForge (SGLang) or Red Hat Speculators
  • Target models: Llama 3.3 70B, Qwen3-Coder-30B MoE, GPT-OSS-120B
  • Quantization: FP8 (Marlin), INT4 AWQ
  • Deployment: Kubernetes + NVIDIA device plugin; HPA on queue-wait metric
  • Eval: ShareGPT, MT-Bench-v2, GSM8K, HumanEval for domain-spread acceptance measurement
  • Reference: TensorRT-LLM speculative decoding for a vendor baseline
Check: Stack0/2
1. What is quantization in this serving stack?Define
2. Why does the server quantize the target with FP8 (Marlin) or INT4 (AWQ)?Explain

Build It

  1. Target model prep. Pick Llama 3.3 70B and quantize it to FP8 via Marlin. Deploy under vLLM 0.7 on 1xH100 (or 2x tensor-parallel).

  2. Draft source. Pull an aligned EAGLE-3 draft head from Red Hat Speculators, or train one with SpecForge. Load it through vLLM's speculative-decoding config.

  3. Baseline numbers. Before any speculation, record tokens/s at batch 1/8/32, p50/p99 latency, and GPU utilization. Publish them.

  4. Enable EAGLE-3. Flip the config and rerun the same benchmark. Report the speedup, the acceptance rate, and the p99 tail-latency delta.

  5. P-EAGLE. Turn on parallel speculation and compare the deeper draft tree against serial EAGLE-3. Report where P-EAGLE starts helping and where it starts to hurt.

  6. Domain traffic. Push ShareGPT, HumanEval, and domain-specific traffic through the same server. Measure acceptance rate per distribution and identify where the drafts drift.

  7. Second target model. Run the same pipeline on Qwen3-Coder-30B MoE. The draft is trickier here because of MoE routing noise. Report what you see.

  8. K8s HPA. Deploy under Kubernetes with HPA tracking queue_wait_ms. Demonstrate scale-out when load triples.

  9. Cost comparison. Compute $/1M tokens against Anthropic Claude Sonnet 4.7 and OpenAI GPT-5.4 on the same eval. Publish it.

Check: Build It0/3
1. Why record tokens/s, p50/p99 and GPU utilization at batch 1/8/32 before enabling any speculation?Explain
2. Why does Build It step 7 call the draft trickier for Qwen3-Coder-30B MoE?Explain
3. What tradeoff should the P-EAGLE comparison against serial EAGLE-3 surface?Explain

Use It

$ curl https://infer.example.com/v1/chat/completions -d '{"messages":[...]}'
[serve]     vLLM 0.7, Llama 3.3 70B FP8, EAGLE-3 active
[decode]    bs=8, accepted_tokens_per_step=3.2, acceptance_rate=0.76
[latency]   first-token 42ms, full-response 980ms (620 tokens)
[cost]      $0.34 per 1M output tokens at sustained throughput
Check: Use It0/2
1. A decode log line reads bs=8, accepted_tokens_per_step=3.2, acceptance_rate=0.76. Which reading is right?Apply
2. Which serving setup does the sample Use It output report as active?Apply

Ship It

outputs/skill-inference-server.md describes the deliverable: a measured serving stack with speculative decoding, a full benchmark report, and a K8s deployment.

WeightCriterionHow it is measured
25Measured speedup vs baseline2.5x+ throughput at matched quality on two models
20Acceptance rate on realistic trafficPer-distribution acceptance-rate report
20P99 tail-latency disciplinep99 at batch 1/8/32 with and without speculation
20OpsK8s deploy, HPA on queue-wait, rollout smooth
15Write-up and methodologyClear explanation of what changed and why
100
Check: Ship It0/3
1. Why does the rubric state the speedup as 2.5x+ at matched quality, not just raw throughput?Explain
2. A submission earns full marks on speedup, acceptance-rate reporting and ops, but publishes no p99 table and a weak write-up that scores zero. What is the score out of 100?Apply
3. A team measures 2.6x on Llama 3.3 70B and 2.4x on Qwen3-Coder-30B, both at matched quality. Is the speedup criterion fully met?Apply
Author's Note

My bet is that the p99 table will decide whether anyone believes this server, so I'd build it before chasing the 2.5x number. A throughput headline is easy to show on friendly traffic, and the Problem section is explicit that tail latency under rejection is worse than without speculation. The rubric splits 40 points between p99 discipline and acceptance rate on realistic traffic, against 25 for the speedup itself.

For the same reason I'd publish the step 3 baseline before touching any speculation config. Every later number, including the p99 delta, only means something if the non-speculative run was measured with the same benchmark.

I'd also pull Exercise 2 into the first deploy instead of treating it as an extra. An ngram fallback that takes over when EAGLE-3 acceptance drops below a threshold is a cheap safety net against the traffic drift step 6 is designed to expose, and I'd want it in place before pointing real traffic at the server.

Exercises

  1. Measure acceptance-rate degradation when the draft is one version behind the target (e.g., Llama 3.3 -> 3.4 drift). Build a monitoring alert.

  2. Implement an ngram fallback: if EAGLE-3 acceptance drops below a threshold, switch to ngram drafts. Report the reliability improvement.

  3. Run a controlled MoE experiment on Qwen3-Coder-30B, with routing noise injected and without. Measure how sensitive draft acceptance is to it.

  4. Extend to H200 (141 GB). Report how much model-size-per-replica headroom you gain and whether you can serve an unquantized Llama 3.3 70B.

  5. Benchmark TensorRT-LLM speculative decoding on the same H100 hardware. Report where it wins against vLLM.

Key Terms

TermWhat people sayWhat it actually means
Draft model"Speculator"Small model that proposes N tokens for the target to verify
EAGLE-3"2026 draft architecture"Draft head trained on target hidden states; ~75% acceptance
P-EAGLE"Parallel speculation"Tree of draft branches verified in one target pass
Acceptance rate"Hit rate"Fraction of drafted tokens accepted without resampling
Quantization"FP8 / INT4"Lower-precision weights to fit more model in GPU memory
Queue wait"HPA metric"Time a request waits in the pending queue before inference starts
Speculators hub"Aligned drafts"Red Hat Neural Magic hub of EAGLE drafts for common open models

Further Reading

Code

Ships With