LearnPython

TensorRT-LLM on Blackwell with FP8 and NVFP4

TensorRT-LLM is NVIDIA-only, and on Blackwell it wins. On GB200 NVL72 with Dynamo orchestration, SemiAnalysis InferenceX measured $0.012 per million tokens on a 120B model in Q1-Q2 2026, against $0.09/M on H100 + vLLM. That is a 7x economic gap. The stack compounds three floating-point regimes: FP8 stays critical for the KV cache and attention kernels because it has the dynamic range they need, NVFP4 (4-bit microscaling) handles weights and activations, and multi-token prediction (MTP) plus disaggregated prefill/decode add another 2-3x on top. Day-0 model support loads FP4 weights directly, with no post-training conversion. The catch is that TRT-LLM is a closed NVIDIA stack, so adopting it trades portability for throughput. Run the math on your own mix of models and hardware before you commit.

TIME: ~75 minutesPREREQUISITES: Phase 17 · 04 (vLLM Serving Internals), Phase 10 · 13 (Quantization)

Learning Objectives

Explain why FP8 stays critical for KV cache and attention even when weights are in NVFP4.
Compute the HBM footprint of a frontier model under BF16, FP8, and NVFP4 and reason about where the savings come from.
Name the Blackwell-specific features TRT-LLM exploits (day-0 FP4, MTP, disaggregated serving, all-to-all primitives).
Decide when TRT-LLM's NVIDIA-lock is worth the 7x cost gap vs vLLM on Hopper.

The Problem

The question behind 2026 inference economics is how many tokens you get per dollar. The answer depends on four stacked choices: hardware generation (Hopper H100/H200 vs Blackwell B200/GB200), precision (BF16 → FP8 → NVFP4), serving engine (vLLM vs SGLang vs TRT-LLM), and orchestration (plain vs disaggregated vs Dynamo).

On Hopper with vLLM, a 120B MoE runs at ~$0.09 per million tokens. On Blackwell with TRT-LLM and Dynamo, the same model runs at ~$0.012, which is 7x cheaper. Part of that gap is hardware, since Blackwell delivers 11-15x the per-GPU LLM throughput of Hopper. The rest comes from the stack: FP4 weights, an MTP draft, disaggregated prefill/decode, and NVLink 5 all-to-all for MoE expert communication.

You cannot replicate this outside NVIDIA's stack, and that is the tradeoff: you give up portability for economics. The goal here is to see which stack choice buys which share of the gap.

Warm-up0/3

1. In a vLLM-style serving engine, what does the KV cache hold for each request?Warm-up

A token 'cat' enters an attention layer and produces the next token; the attention layer also reads an unlabelled box built from earlier tokens. — Illustrative single decode step.

2. In quantization, why does giving each small block of weights its own scale factor help at 4 bits?Warm-up

Eight weight values in one block with one outsized value marked, under one unknown shared scale factor for the whole block. — Illustrative values; a real block holds far more than eight.

3. A 10-billion-parameter model has its weights re-stored at 8 bits each instead of 16 bits each. What happens to the weight memory?Warm-up

The same 10 billion weights stored at 16 bits each and at 8 bits each, each with an unknown total size in GB. — Setup; parameter count and bit widths are the question's own numbers.

Check: The Problem0/2

1. Roughly what cost gap per million tokens separates Blackwell + TRT-LLM + Dynamo from H100 + vLLM on a 120B model?Define

A fixed GB200 NVL72 price of $0.012 per million tokens, next to an unknown H100 + vLLM price and an unknown gap between the two. — The GB200 figure is given; the H100 figure and the gap are the question's own unknowns.

2. Why can't you get the full Blackwell cost advantage just by buying Blackwell GPUs and keeping the same serving setup?Explain

A hardware fact (11 to 15x per-GPU throughput) listed above four separate stack techniques: FP4 weights, MTP draft, disaggregation and NVLink 5 all-to-all. — Facts from the lesson; not yet combined into one explanation.

The Concept

Why FP8 is still the floor for KV cache

It is easy to assume NVFP4 applies everywhere in 2026. It does not. The KV cache needs FP8 (8-bit floating point) because it stores attention keys and values that span a wide dynamic range. Quantize KV to FP4 and accuracy loss is catastrophic: the tail of the distribution drops off and attention scores collapse. FP8's exponent bits give the KV cache the range it needs.

NVFP4 (2025-2026) applies to weights and activations. It uses microscaling, so each block of weights carries its own scale factor and small blocks can cover different dynamic ranges without the loss of a single per-tensor scale. For activations, FP4 holds up because activations stay in a small range within a layer.

The typical Blackwell config:

Weights: NVFP4 (4-bit microscaling).
Activations: NVFP4.
KV cache: FP8.
Attention accumulator: FP32 (softmax stability).

The Blackwell-specific primitives TRT-LLM uses

Day-0 FP4 weights: model providers ship FP4 weights directly and TRT-LLM loads them without post-training conversion. There is no AWQ / GPTQ step for FP4.
Multi-token prediction (MTP): the same idea as EAGLE (Phase 17 · 05), integrated into the TRT-LLM build.
Disaggregated serving: prefill and decode run on separate GPU pools, with KV cache transferred over NVLink or InfiniBand. Same idea as Dynamo (Phase 17 · 20).
All-to-all communication primitives: NVLink 5 cut MoE expert communication latency by 3x vs Hopper, and TRT-LLM's MoE kernels are tuned for it.
NVFP4 + MXFP8 microscaling: scale-factor handling is hardware-accelerated on Blackwell Tensor Cores.

The numbers worth keeping

HGX B200 at $0.02/M tokens on GPT-OSS-120B via TRT-LLM.
GB200 NVL72 at $0.012/M tokens via Dynamo (orchestrating TRT-LLM).
H100 + vLLM ≈ $0.09/M tokens on a comparable workload.
2.8x throughput gain from three months of TRT-LLM updates (2026).
11-15x per-GPU LLM throughput, Blackwell vs Hopper.
MLPerf Inference v6.0 (April 2026): Blackwell dominated every submitted task.

What FP4 actually costs in quality

NVFP4 is aggressive. On reasoning-heavy workloads (chain-of-thought, math, code-gen with long context), FP4 weights degrade visibly. Per-block calibration mitigates the damage but does not remove it. Teams shipping reasoning models often use FP8 weights with FP4 activations as a compromise, or stay on H200 with FP8 throughout.

My rule: validate task quality on your own eval set before committing to NVFP4 weights.

Why this is an NVIDIA-lock decision

TRT-LLM is C++ and CUDA with closed-source kernels, and models are compiled for a specific GPU SKU. AMD, Intel, and ARM are out. If your infra strategy is multi-vendor, TRT-LLM is a non-starter for the tier it would serve, though you can still run vLLM on mixed hardware. If you are NVIDIA-only, the 7x gap pays for the lock.

2026 practical recipe

With a $100M+ annual inference bill, staying on Hopper + vLLM leaves 7-10x on the table. Migrate the cost-dominant workloads to Blackwell + TRT-LLM + Dynamo. Keep the experimentation tier on H100 + vLLM so model iteration stays fast. Validate quality on each NVFP4-converted model before it reaches production.

The disaggregation bonus

TRT-LLM's disaggregated serving (separate prefill and decode pools) is covered in depth in Phase 17 · 20. On Blackwell the multipliers stack: FP4 weights × MTP speedup × disaggregated placement × cache-aware routing. The 7x number assumes the full stack.

Check: The Concept0/21

1. Which Blackwell feature lets TRT-LLM load a model without a post-training conversion step?Define

Both paths' structure comes from the lesson's description of day-0 weight loading.

2. What is MTP in TRT-LLM?Define

A baseline decode step producing one token per pass, next to an unnamed technique where the big model checks several candidate tokens in one pass, sometimes yielding more than one new token. — Illustrative decode steps; no technique is named in the picture.

3. Which primitive runs prefill and decode on separate GPU pools and moves KV cache between them?Define

Two separate GPU pools labelled prefill and decode, with KV cache moving from one to the other, and an unknown box naming this primitive. — The two pools and the KV-cache transfer are the question's own setup.

4. What is FP8, and what role does it play in a typical Blackwell setup?Define

The bit layout of one FP8 value: eight bit cells split into one sign bit, four exponent bits and three mantissa bits. — FP8 (E4M3) bit layout.

5. What is NVFP4?Define

Sixteen 4-bit value cells in one block, all sharing one scale factor drawn below them, labelled as belonging only to this block. — Generic microscaling layout; block size is illustrative.

6. What is MXFP8?Define

Eight 8-bit value cells in one block sharing one scale factor drawn below them, labelled as belonging only to this block. — Generic microscaling layout; block size is illustrative.

7. What is the all-to-all pattern in MoE serving?Define

Tokens leaving one GPU fan out to three separate expert GPUs, each receiving the tokens that picked its expert. — Schematic MoE routing; counts are illustrative.

8. What is InferenceX?Define

A benchmark report table with column headings for model, hardware, engine and cost per million tokens, and every cell left as an unknown value. — Illustrative headings only; no figures shown.

9. Which precision combination is the typical Blackwell configuration?Define

Four boxes for one attention layer: weights, activations, KV cache and attention accumulator, each with an unknown precision. — Setup; all four precisions are left blank.

10. By how much did NVLink 5 cut MoE expert-communication latency versus Hopper?Define

A labelled bar for Hopper's MoE expert-communication latency next to a dashed, unlabelled box for Blackwell's, whose height is unknown. — Illustrative Hopper bar; Blackwell's bar height is hidden.

11. Why keep the KV cache in FP8 rather than NVFP4 on Blackwell?Explain

A curve over key and value magnitudes for one attention head, mostly bunched near zero with a long low tail stretching out to one side. — Typed illustration of a dynamic-range shape; not a real model's data.

12. Why does the typical config keep the attention accumulator in FP32 even though the weights are NVFP4?Explain

Six attention scores on a row, one far larger than the rest, feeding a softmax whose running-sum precision is left unknown. — Illustrative softmax scores.

13. Why does FP4 hold up for activations when it fails for the KV cache?Explain

A narrow, tightly bunched curve of activation values within one layer, with an unknown box asking whether 4 bits can represent them. — Typed illustration of a narrow activation range.

14. What is the dominant tradeoff of choosing the TRT-LLM stack?Explain

A model going through a closed-kernel TRT-LLM compile step into an engine binary compiled for one unnamed GPU SKU. — The compile-to-one-SKU step is stated directly in the Concept section.

15. In disaggregated serving, what has to cross from the prefill pool to the decode pool for each request?Explain

A prefill pool and a decode pool with an unknown payload crossing between them for one request. — Setup; the payload that crosses is the question's own unknown.

16. Under an FP8 KV cache, what happens to the HBM footprint when weights go from BF16 to NVFP4?Explain

Weight precision at BF16 and at NVFP4, plus the KV cache held fixed at FP8, each with an unknown GB total. — Setup; weight and KV footprints are left as unknowns.

17. NVFP4 conversion drops your reasoning model's MATH score by 3 points. What is the best next step?Analyze

A reasoning model's MATH score dropping 3 points after NVFP4 weight conversion, with an unknown box for the best next step. — Setup; the drop is given, the next step is the question's own unknown.

18. A team moves its KV cache from FP8 to NVFP4 to save HBM, and attention quality drops sharply. What is the likely cause?Analyze

A team's KV cache moved from FP8 to NVFP4 to save HBM, with attention quality dropping sharply and an unknown likely cause. — Setup; both facts are given in the question.

19. A team keeps its H100s, swaps vLLM for TRT-LLM, and expects the 7x gap. Why will they likely fall short?Analyze

H100 hardware stays unchanged while the serving engine switches from vLLM to TRT-LLM, with an unknown resulting gap. — Setup; the GPU generation and engine swap are the question's own premise.

20. Your company runs mixed NVIDIA and AMD fleets and iterates on models weekly. What is the sound plan?Design

A fleet fact (NVIDIA plus AMD GPUs) and an iteration fact (weekly model changes), with an unknown box for the serving plan. — Setup; both facts are given in the question.

21. You must serve a long chain-of-thought math model on Blackwell, accuracy matters more than the last bit of cost, and you have no eval results yet. Which plan fits?Design

A long chain-of-thought math model where accuracy matters more than the last bit of cost and no evals have run yet, with an unknown precision plan. — Setup; all three facts are given in the question.

Use It

code/main.py computes HBM footprint, decode throughput (memory-bound regime), and $/M tokens across five stacks: H100 + BF16 + vLLM, H100 + FP8 + vLLM, H200 + FP8 + vLLM, B200 + NVFP4/FP8 + TRT-LLM, and GB200 NVL72 + TRT-LLM + Dynamo. It runs four model shapes through them. Run it to see the compounding effect and how much each change adds to the gap.

Check: Use It0/7

1. In the toy script, why does changing the KV cache precision leave tokens per second unchanged?Explain

The decode_throughput function's three lines, computing bytes per token from weight_bits only, with no kv_bits term anywhere. — Quoted from the lesson's main.py.

2. Using the script's formula, how many GB of weights does a 120B-parameter model need at NVFP4 (4 bits per weight)?Apply

A 120-billion-parameter model at 4 bits per weight, next to the script's weight-footprint formula and an unknown GB total. — Setup; the formula is given, the result is the question's own unknown.

3. The script runs the 120B MoE (36B active) on H100 + FP8 with 3.35 TB/s HBM. Roughly what does decode_throughput return per GPU?Apply

A 36-billion active-parameter model at 8-bit weights on 3.35 TB/s HBM, with an unknown tokens-per-second result. — Setup; figures are the question's own numbers.

4. For the same 120B MoE on the B200 stack, you set mtp_factor to 1.0 and change nothing else. Roughly what throughput results?Apply

The real B200 row with mtp_factor forced to 1.0 and disagg_factor left at 1.6, with an unknown tokens-per-second result. — Setup; only mtp_factor is changed here.

5. The script runs a dense 405B model on the B200 + NVFP4 stack (192 GB HBM). What does that row show?Apply

A 405-billion-parameter dense model at 4 bits per weight against a 192 GB single-GPU HBM budget, with an unknown weight size and fit result. — Setup; figures are the question's own numbers.

6. The GB200 NVL72 stack uses the same 8 TB/s HBM, NVFP4 and mtp_factor 1.8 as B200, but disagg_factor 2.5 instead of 1.6. For the 120B MoE (36B active), roughly what tokens/s?Apply

The GB200 NVL72 stack's own stated factors, matching B200 except for a disagg_factor of 2.5, with an unknown tokens-per-second result. — Setup; figures are the question's own numbers.

7. The script prints about $1.04 per million tokens for B200, while the lesson quotes $0.02 for HGX B200. A teammate says this proves the TRT-LLM claim is false. What is wrong with that conclusion?Analyze

The script's own $1.04 per million tokens next to the lesson's quoted $0.02, with an unknown box asking what's wrong with calling this proof the claim is false. — Setup; both dollar figures are given in the question.

Ship It

The output is outputs/skill-trtllm-blackwell-advisor.md. Give it a workload, a model size, and an annual token volume, and it decides whether the Blackwell + TRT-LLM stack is worth the NVIDIA-lock.

Author's Note

I'd run main.py and read the 120B MoE block first. Going from H100 + FP8 to B200 + NVFP4/FP8 takes decode from 93 to 1280 tokens/sec per GPU, about 14x, and that jump is the product of four constants on the Stack dataclass: HBM bandwidth, weight bits, mtp_factor, and disagg_factor. They are hand-set values, not measurements. The B200 row prints about $1.04 per million tokens, far from the $0.012 quoted above, so read the script for the shape of the compounding and not for a price.

Two simplifications are worth knowing before you trust it. Decode throughput counts only the weight bytes read per token, so KV precision changes the footprint column and leaves tok/s alone. And there is no quality term anywhere. That is the part I'd worry about most with NVFP4 on reasoning workloads, which is why my bet is that an eval run on your own model matters more than any multiplier in the script.

Exercises

Run code/main.py. On a 120B MoE with 30% active parameters, compute the memory-bandwidth-limited decode throughput on H100 BF16, H100 FP8, and B200 NVFP4/FP8. Where does the biggest jump come from?
A customer spends $2M/year on H100 + vLLM. What is the break-even number of Blackwell GPUs they need to buy to amortize a migration to TRT-LLM in 12 months, given the 7x economic gap?
You see accuracy drop 3 points on MATH after NVFP4 weight conversion. Name two recovery paths: one quality-first (keep FP8 weights), one cost-first (calibrate with in-domain data).
Read the MLPerf v6.0 inference results. Which task has the smallest Blackwell-over-Hopper gap, and why?
Compute the HBM needed for a 405B model at NVFP4 weights + FP8 KV cache at 128k context. Does it fit on a single GB200 NVL72 node?

Key Terms

Term	What people say	What it actually means
FP8	"eight-bit float"	8-bit floating point; used for KV cache and attention due to dynamic range
NVFP4	"four-bit micro"	NVIDIA's 4-bit microscaling FP format; weights and activations on Blackwell
MXFP8	"MX eight"	Microscaling FP8 variant; hardware-accelerated on Blackwell Tensor Cores
Day-0 FP4	"ship FP4 weights"	Model providers release weights already in FP4; no post-train conversion step
MTP	"multi-token prediction"	TRT-LLM's integrated speculative-decoding draft (Phase 17 · 05)
Disaggregated serving	"split prefill/decode"	Prefill and decode on separate GPU pools; KV transferred over NVLink/IB
All-to-all	"MoE expert comm"	Communication pattern routing tokens to expert GPUs; NVLink 5 cuts 3x
InferenceX	"SemiAnalysis inference bench"	The 2026 industry-accepted cost-per-token benchmark