EAGLE-3 Speculative Decoding in Production

Decode is memory-bound. On an H100 running Llama 3.3 70B FP8, each decoded token reads ~140 GB/s of weights and emits one token. The GPU compute sits mostly idle during decode, because the bottleneck is HBM bandwidth and not matmul throughput.

Speculative decoding exploits that gap. A cheap draft model generates K candidate tokens, then the target model verifies all K in a single forward pass. A verified token is nearly free, since it is amortized into a K-wide forward the target would have had to run anyway.

The classic approach uses a smaller model from the same family (Llama 3.2 1B drafting for Llama 3.3 70B). It works, but acceptance rate is mediocre because the small model's distribution diverges from the target's. EAGLE, then EAGLE-2, then EAGLE-3 train a light draft head directly on the target model's internal states, so the draft tracks the target much more closely. That is why alpha goes from 0.4 with a draft model to 0.6-0.8 with EAGLE-3.

The catch is that EAGLE-3 is opt-in in vLLM 2026. You have to set speculative_config explicitly, and without it you get no acceleration. Teams that flip it on without measuring alpha on their real traffic often see tail latency get worse, not better.

What speculative decoding actually buys

Without spec decode, each token costs one target forward. With draft length K and acceptance rate alpha, the expected number of tokens per target forward is 1 + K * alpha. The speedup is (1 + K * alpha) / (1 + epsilon), where epsilon is the draft-plus-verify overhead. For K=5, alpha=0.7: (1 + 5*0.7) / (1 + 0.1) = 4.5 / 1.1 = 4.1x. Real-world numbers cluster around 2-3x, because alpha is rarely that high on production traffic and epsilon grows at high batch size.

Why alpha comes first

Rejected tokens do not disappear. They force a second target forward for the first rejected token. On a workload where alpha drops to 0.4, you pay draft overhead, then verification, then the re-roll. At high concurrency (say 256 concurrent) the decode batch is already large enough that the memory-bandwidth gap between "target alone" and "target with verify" shrinks. Below alpha 0.55 on most 2026 hardware, spec decode is net negative.

Alpha varies by workload. On ShareGPT-style general chat, EAGLE-3 trained on ShareGPT hits 0.6-0.8. On domain-specific traffic (code, medical, legal) a draft head trained on general data drops to 0.4-0.6. Training a domain-specific draft head recovers alpha, and it is a light, quick job compared to finetuning the target.

EAGLE generations at a glance

• Classic draft model: a small model of the same family. Alpha 0.3-0.5. The infrastructure is simple: two models loaded, and the draft runs K forwards per target forward.
• EAGLE-1 (2024): a single draft head trained on the target's last-layer hidden states. Alpha ~0.5-0.6, with a small parameter overhead on top of the target.
• EAGLE-2 (2025): adaptive draft length and tree-based drafts, so multiple branches get verified in one target pass. Alpha ~0.6-0.7, at the price of a more complex draft scheduler.
• EAGLE-3 (2025-2026): a draft head trained on multiple target layers instead of just the last, which gives better alignment. Alpha ~0.6-0.8 on general chat.

The 2026 production recipe

1. Ship the target model plain. Measure baseline TTFT, ITL, and throughput at your target concurrency.
2. Enable the EAGLE-3 draft via vLLM speculative_config and re-run the benchmark.
3. Log acceptance rate alpha. vLLM V1 reports it as spec_decode_metrics.accepted_tokens_per_request; divide by the requested draft length to get alpha.
4. If alpha < 0.55 on your production traffic distribution, disable spec decode or train a domain-specific EAGLE-3 draft.
5. Re-run at production concurrency and confirm P99 ITL did not get worse.

The production pitfall: P99 tail

Mean ITL drops with spec decode, but P99 can get worse if you do not tune. A rejected draft triggers a two-pass sequence (draft, verify-fail, reroll), and under a full batch those two passes serialize. Watch P99 ITL, not P50.

Where EAGLE-3 is already deployed

Google deployed speculative decoding in AI Overviews in 2025 (same quality, faster response). vLLM V1 ships speculative_config as the documented interface, and N-gram GPU speculative decoding in V1 is the variant that works with chunked prefill. SGLang supports EAGLE-3 as the recommended draft path for prefix-heavy workloads.

Break-even math in one line

Expected speedup is S(alpha, K) = (1 + K*alpha) / (1 + verify_overhead). Set S = 1 and solve for alpha: alpha_breakeven = verify_overhead / K. With a typical verify_overhead of ~0.15 and K=5, that gives alpha_breakeven = 0.03. But that is the raw decode math. At high concurrency the verify overhead rises and the decode batch already amortizes memory reads across sequences, so the effective alpha_breakeven climbs to ~0.45-0.55 in practice.

When not to use speculative decoding

• Batch-1 offline generation where latency does not matter. Use the plain target.
• Very short outputs (under 50 tokens). Draft overhead and verify cost dominate.
• Specialized domains without a domain-trained draft head. Alpha is too low.
• vLLM v0.18.0 with draft-model spec decode and --enable-chunked-prefill. This combination does not compile. The documented exception is N-gram GPU spec decode in V1.

code/main.py simulates a decode loop with and without speculative decoding across a range of alpha values and draft lengths K. It prints the break-even alpha, the speedup, and the tail behavior. Run it on a few (alpha, K) combinations and see where speculative decoding stops paying.

This lesson produces outputs/skill-eagle3-rollout.md. Give it a target model, a traffic distribution description, and a concurrency target, and it returns a staged EAGLE-3 rollout plan: benchmark the baseline, enable the config, measure alpha, gate on alpha >= 0.55, and watch P99 ITL.

1. Run code/main.py. At K=5, what alpha do you need for a 2x speedup? For a 3x speedup? How sensitive is that to verify_overhead?
2. Imagine production traffic splits 70% general chat, 30% code. General chat hits alpha 0.7 with EAGLE-3 trained on ShareGPT; code hits alpha 0.4. What is blended alpha and is spec decode net-positive?
3. Read the vLLM speculative_config documentation. Name the three modes (draft model, EAGLE, N-gram) and which one is compatible with chunked prefill.
4. You see mean ITL drop 25% after enabling EAGLE-3 but P99 ITL went up 15%. Diagnose and propose a mitigation.
5. Compute the memory cost of the EAGLE-3 draft head for Llama 3.3 70B. How does it compare to running Llama 3.2 1B as a classic draft?

• vLLM Speculative Decoding docs: authoritative source on speculative_config and chunked-prefill compatibility in V1.
• vLLM Speculative Config API: the exact field set.
• EAGLE paper (arXiv:2401.15077): the original EAGLE draft-head formulation.
• EAGLE-2 paper (arXiv:2406.16858): adaptive drafts and trees.
• UC Berkeley EECS-2025-224: efficient LLM system with speculative decoding.
• BentoML Speculative Decoding: production rollout checklist.