LearnPython

Production Quantization — AWQ, GPTQ, GGUF K-quants, FP8, MXFP4/NVFP4

Quantization format is a function of hardware, serving engine, and workload, so there is no universal pick. GGUF Q4_K_M or Q5_K_M owns CPU and edge, delivered through llama.cpp and Ollama. GPTQ wins inside vLLM when you need multi-LoRA on the same base. AWQ with Marlin-AWQ kernels delivers ~741 tok/s on a 7B class model with the best Pass@1 at INT4, which makes it the 2026 default for datacenter production. FP8 is the middle ground on Hopper, Ada, and Blackwell: near-lossless and widely supported. NVFP4 and MXFP4 (Blackwell microscaling) are aggressive and need per-block validation. Two traps bite teams. The calibration dataset has to match the deployment domain, and the KV cache is separate from weight quantization, so "my model is 4 GB now" forgets the tens of GB of KV cache at production batch sizes.

TIME: ~75 minutesPREREQUISITES: Phase 10 · 13 (Quantization foundations), Phase 17 · 04 (vLLM Serving Internals)

Learning Objectives

Name the six production quantization formats and their sweet spots in 2026.
Pick a format given hardware (CPU vs GPU, Hopper vs Blackwell), engine (vLLM, TRT-LLM, llama.cpp), and workload (routine chat, reasoning, multi-LoRA).
Compute the weight memory saved and the KV cache left untouched for a chosen format.
Name the calibration-dataset pitfall that degrades quantized models on domain traffic.

The Problem

Quantization cuts memory and HBM bandwidth, which is exactly what decode needs. An FP16 70B model is 140 GB of weights. Quantize the weights to INT4 (AWQ or GPTQ) and the model is 35 GB. That fits in one H100 with some room left for KV cache, and the room matters: at 128 concurrent sequences with 2k context, code/main.py puts KV cache alone at 68.7 GB for a 70B-class model.

Quantization costs something, though. Aggressive formats degrade quality, especially on reasoning-heavy tasks. Different formats work with different engines, and different hardware supports different precisions natively. The 2026 format zoo is real, and you can't copy someone else's choice. You pick based on your own stack.

Warm-up0/3

1. Decode on a GPU is mostly limited by which resource, and why does quantizing weights help it?Warm-up

Four labeled boxes in a row, Disk, PCIe bus, HBM memory, and Tensor Cores, connected by plain arrows showing the path data takes into a GPU. None of the four is marked or colored differently. — The four candidates from the options, drawn as a plain, unlabeled data path.

2. In a vLLM server, KV cache memory grows with which quantities?Warm-up

Three separate chat sessions on one server, each drawn as a named box followed by a row of small message tiles of different lengths. No counts, axes, or formulas are labeled. — Several ongoing conversations sharing one server.

3. A model has 7 billion parameters stored at 16 bits (2 bytes) each. About how much memory do the weights alone take?Warm-up

A card with two given rows, 7,000,000,000 parameters and 2 bytes per parameter, above a third row for weight memory left as a question mark. — The two numbers from the question, multiplication left undone.

Check: The Problem0/2

1. A team plans to serve a 70B model with AWQ INT4 on one H100 80GB at 128 concurrent sequences and 2k context, budgeting 35 GB of weights plus a little overhead. What does the toy in main.py show?Apply

A card titled 70B model, 128 concurrent, 2k context, listing budget H100 80GB capacity 80 GB, given AWQ-Int4 weights 35 GB, KV cache FP16 separate from weights as a question mark, and total as a question mark. — The one number given in the question, with KV cache and the total left to work out.

2. For a 70B model at 128 concurrent and 2k context in main.py, switching from the BF16 baseline to AWQ-Int4 saves how much weight memory and how much KV cache?Apply

Two separate things to work out, the new weight size, and whether KV cache's bit width changes at all.

The Concept

The six formats

Format	Bits	Sweet spot	Engines
GGUF Q4_K_M / Q5_K_M	4-5	CPU, edge, laptops	llama.cpp, Ollama
GPTQ	4-8	Multi-LoRA on vLLM	vLLM, TGI
AWQ	4	Datacenter GPU production	vLLM (Marlin-AWQ), TGI
FP8	8	Hopper/Ada/Blackwell datacenter	vLLM, TRT-LLM, SGLang
MXFP4	4	Blackwell multi-user	TRT-LLM
NVFP4	4	Blackwell multi-user	TRT-LLM

GGUF: the CPU/edge default

GGUF is a file format rather than a quantization scheme. It bundles the K-quant variants (Q2_K, Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, Q8_0) in one container. Q4_K_M and Q5_K_M are the production defaults, with near-BF16 quality at 4-5 bits. It is the best choice for CPU or edge serving because llama.cpp is by far the fastest CPU inference engine.

The cost shows up in vLLM, where GGUF runs at ~93 tok/s on 7B because the format isn't optimized for GPU kernels. Use GGUF when the deployment target is CPU or edge, and pick something else on a GPU.

GPTQ: multi-LoRA in vLLM

GPTQ is a post-training quantization algorithm with a calibration pass. Marlin kernels make it fast on GPU (2.6x speedup vs non-Marlin GPTQ), at ~712 tok/s on 7B.

Its distinguishing win is that GPTQ-Int4 supports LoRA adapters in vLLM. If you serve a base model plus 10-50 fine-tuned variants, each as a LoRA, GPTQ is your path. NVFP4 does not support LoRA yet as of early 2026.

AWQ: the datacenter GPU default

Activation-aware Weight Quantization protects the ~1% most-salient weights during quantization. With Marlin-AWQ kernels it gets a 10.9x speedup vs naive and ~741 tok/s on 7B, with the best Pass@1 among INT4 formats.

Pick AWQ for new GPU serving unless you need multi-LoRA (GPTQ) or aggressive Blackwell FP4 (NVFP4).

FP8: the reliable middle

8-bit floating point is near-lossless and widely supported. Hopper Tensor Cores accelerate it natively, and Blackwell inherits that. FP8 is the safe 2026 default when quality matters most (reasoning, medical, code-gen). FP8 weights take twice the memory of INT4 weights, but the quality risk is far lower.

MXFP4 / NVFP4: Blackwell aggressive

These are microscaling FP4 formats, where each block of weights has its own scale factor. They are aggressive but hardware-accelerated on Blackwell Tensor Cores. They halve the bytes per token versus FP8, which is the economic win in Phase 17 · 07.

Caveats:

No LoRA support yet (early 2026).
Quality drop visible on reasoning-heavy workloads.
Validate on your eval set per model.

The calibration trap

AWQ and GPTQ need a calibration dataset, typically C4 or WikiText. For domain models (code, medical, legal), calibrating on generic web text lets the algorithm make wrong decisions about which weights to protect. Pass@1 on HumanEval can drop several points.

The fix is to calibrate on in-domain data. Hundreds of domain samples is usually enough. Test on the eval set before shipping.

The KV cache trap

AWQ shrinks weights to 4 bits. KV cache is separate and stays at FP16/FP8. For a 70B model with AWQ:

Weights: ~35 GB (INT4 from 140 GB).
KV cache at 128 concurrent × 2k context (FP16 KV): ~69 GB.
Activations: ~3.5 GB.
Total: ~107 GB, which does not fit an H100 80GB and lands on an H200 141GB.

Quantizing the KV cache to FP8 and the weights to NVFP4 brings the same setting to ~73 GB, which does fit one H100. "I quantized my model to 4 GB" forgets the KV cache, which is tens of GB at production batch sizes. Budget HBM for the whole thing.

KV cache quantization (FP8 KV or INT8 KV) is a separate choice with its own tradeoffs. It affects attention accuracy directly, so it isn't a free win.

AWQ INT4 is hazardous for reasoning

Chain-of-thought, math, and long-context code-gen suffer visibly from aggressive quantization. AWQ INT4 loses ~3-5 points on MATH. For reasoning-heavy workloads, ship FP8 or BF16 and accept the memory cost.

2026 picking guide

CPU/edge serve: GGUF Q4_K_M.
GPU serve, routine chat, no LoRA: AWQ.
GPU serve, multi-LoRA: GPTQ with Marlin.
Reasoning workload: FP8.
Blackwell datacenter, validated quality: NVFP4 + FP8 KV.
Ambiguous: run a 1,000-sample eval on each candidate format.

Check: The Concept0/25

1. What is GGUF?Define

A box labeled GGUF file containing six small tagged blocks, Q2_K, Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, Q8_0. Two arrows lead down from it to boxes labeled llama.cpp and Ollama. — The six variant tags inside one GGUF file, and the two engines that load it.

2. What does Q4_K_M name?Define

The tag Q4_K_M with three lines pointing down to three labeled parts, Q4 as bit width, K as quant scheme, and underscore M as size setting. A note below lists the other GGUF tags without expanding them. — The tag Q4_K_M split into the three parts its name encodes.

3. What is GPTQ?Define

The calibration pass that produces GPTQ-Int4, sitting inside a vLLM deployment.

4. What does "activation-aware" mean in Activation-aware Weight Quantization?Define

A grid of 24 small weight cells labeled 4-bit. Three scattered cells have a different border and are labeled high precision instead. — A small set of weights kept at higher precision while the rest drop to 4 bits. Not to scale: the real figure is about 1 percent.

5. What are Marlin kernels?Define

A flow of three boxes, INT4 weights labeled GPTQ or AWQ, into Marlin CUDA kernel, into GPU Tensor Cores. — Where a Marlin kernel sits between quantized weights and the GPU's compute units.

6. What is FP8 in the lesson's format picking guide?Define

FP8's hardware path. No calibration step is drawn here.

7. What makes MXFP4 and NVFP4 "microscaling" formats?Define

One weight tensor split into four blocks. Each block has its own label, scale s1 through s4, and contains a small grid of cells marked 4-bit cells. — Each block carries its own scale factor instead of one scale for the whole tensor.

8. What is a calibration dataset used for in AWQ and GPTQ?Define

A flow of four boxes, sample text, C4 or WikiText, into run through the model, into pick which weights to protect, into AWQ or GPTQ quantizer. — What a calibration pass does with its input text.

9. What is KV cache quantization?Define

Two side-by-side boxes separated by a dashed line. Left reads weights, quantized, for example INT4. Right reads KV cache, FP8 or INT8, chosen separately. — Two separate precision choices in the same deployment.

10. Why is GGUF Q4_K_M the production default for CPU and edge serving?Explain

Three connected boxes, Target CPU or edge, Engine llama.cpp, Format GGUF Q4_K_M 4 to 5 bit. — The deployment target, engine, and format named in the question.

11. On a 7B-class model, what throughput does each format reach, in order GGUF in vLLM, GPTQ with Marlin, and AWQ with Marlin-AWQ?Explain

Three bars of equal height on a 7B-class model, labeled GGUF in vLLM, GPTQ with Marlin in vLLM, and AWQ with Marlin-AWQ in vLLM. Each bar is marked with a question mark instead of a number. — Same height on purpose. The question asks you to recall the real numbers.

12. Why does calibrating AWQ on generic web text hurt a domain model?Explain

The mismatch the question asks about, left unresolved.

13. What speedup do Marlin kernels give GPTQ, and Marlin-AWQ kernels give AWQ?Explain

Two boxes, GPTQ with Marlin and AWQ with Marlin-AWQ, each tagged with a question mark times symbol and the words versus non-Marlin. — Two kernels, two unknown speedup factors.

14. Why does quantizing a 70B model's weights to 4 bits leave the KV cache size unchanged?Explain

An outer box labeled HBM containing two separate boxes, Weights store, INT4, quantized, and KV cache store, FP16 or FP8, holds per-token keys and values. — Two separate memory regions inside the same GPU.

15. What happens to chain-of-thought and math workloads under AWQ INT4, and what does the lesson ship instead?Explain

A box labeled AWQ INT4 above two workload boxes, routine chat, and reasoning slash math chain-of-thought. The reasoning box has a label reading quality, question mark, below it. — Same weight format, two different workload types. The gauge is left blank.

16. How do MXFP4 and NVFP4 change weight memory relative to FP8?Explain

Two boxes side by side, FP8 at 8 bits per weight, and MXFP4 or NVFP4 at 4 bits per weight. No ratio or percentage is written. — The two raw bit widths side by side.

17. Why does the lesson tie MXFP4 and NVFP4 to Blackwell datacenter GPUs?Explain

A flow of three boxes, NVFP4 or MXFP4 model weights, into Blackwell GPU, into TRT-LLM. — The pairing of format, GPU generation, and engine named in the lesson.

18. A team serving a base model with many LoRA variants wants NVFP4 for the memory savings. What blocks them as of early 2026?Analyze

A base model box connected to a many LoRA variants box, and a separate NVFP4 format box. A dashed line between the LoRA variants box and the NVFP4 box carries a question mark. — The team's plan, with the compatibility question left open.

19. A code-generation model quantized with AWQ, calibrated on C4, scores several points lower on HumanEval Pass@1 than BF16. Which fix targets the cause?Analyze

A card, model code generation, calibrated on C4 generic web text, measured HumanEval Pass at 1 dropped, fix left as a question mark. — The scenario from the question, with the fix left for you to pick.

20. A team says "we quantized the model to 4 GB, so it fits," yet the server runs out of HBM at 128 concurrent sequences. Which memory item did they leave out?Analyze

A card, team's math 4 GB weights only, server at 128 concurrent out of memory, missing left as a question mark. — The gap between the team's number and what actually ran out.

21. A team loads a GGUF Q4_K_M model in vLLM on an H100 and measures ~93 tok/s on 7B, while AWQ with Marlin-AWQ gets ~741. What is the diagnosis?Analyze

Two bars on the same H100, same vLLM engine, GGUF Q4_K_M at 93 tokens per second, and AWQ with Marlin-AWQ at 741 tokens per second. — The two measurements from the question, same hardware and engine for both.

22. You serve one base model plus 30 fine-tuned LoRA variants in vLLM on datacenter GPUs. Which format do you pick?Design

A card, base model plus 30 LoRA variants, serving vLLM, hardware datacenter GPU, format left as a question mark. — The scenario from the question, format left for you to pick.

23. You run a reasoning-heavy workload on H100s where quality is non-negotiable and memory is available. Which weight precision do you ship?Design

A card, workload reasoning chain-of-thought or math, hardware H100, priority quality non-negotiable, memory available, weight precision left as a question mark. — The four conditions from the question, precision left for you to pick.

24. You must serve a 70B chat model at 128 concurrent, 2k context on one H100 80GB with no LoRA need. Judging by the memory budget in the toy, which stack fits, and what must you do before shipping?Design

A card, target H100 80GB capacity 80 GB, model 70B chat 128 concurrent 2k context, LoRA needed no, stack left as a question mark, total left as a question mark. — The scenario from the question, stack and total left blank.

25. A new workload mixes chat and code on GPUs, quality tolerance is unclear, and no rule in the picking guide clearly applies. What does the lesson recommend?Design

A card, workload mixed chat and code, quality tolerance unclear, picking guide no rule clearly matches, next step left as a question mark. — The scenario from the question, next step left for you to pick.

Use It

code/main.py computes memory footprint (weights + KV + activations) and relative throughput across the six formats for a range of model sizes. It shows where KV cache dominates, where weight compression pays, and where FP8 is the safe pick.

Check: Use It0/5

1. In main.py, a 70B model with FP8 weights and FP8 KV runs at 128 concurrent and 2k context (weights 70 GB, KV about 34.4 GB, activations 3.5 GB). Which GPU does gpu_check return?Apply

A legend of four GPU tiers by total GB, up to 80 is H100 80GB, up to 141 is H200 141GB, up to 192 is B200 192GB, above that is MULTI-GPU, none highlighted. Below it, a note giving weights 70 GB, KV cache 34.4 GB, activations 3.5 GB, with total left as a question mark. — The gpu_check thresholds and the three numbers from the question, sum left blank.

2. main.py computes weight_gb = params_b * weight_bits / 8. For the 405B model at 4 bits, what weight memory does it print?Apply

A card with the formula weight_gb equals params_b times weight_bits divided by 8, then params_b 405 and weight_bits 4, with weight_gb left as a question mark. — The formula and its two inputs for the 405B, 4-bit case.

3. In main.py, relative_throughput returns 16 divided by weight_bits. What rel tput does it print for FP8 and for GGUF Q5_K_M?Apply

A card with the formula relative_throughput equals 16 divided by weight_bits, then two rows, FP8 weight_bits 8, result a question mark times, and GGUF Q5_K_M weight_bits 5, result a question mark times. — The formula and the two bit widths from the question, results left blank.

4. The 70B scenario at 128 concurrent and 2k context has 68.7 GB of FP16 KV. main.py's third scenario uses 256 concurrent and 8k context. Roughly what KV does it print?Apply

A card, 128 concurrent 2k context given at 68.7 GB, scale concurrency 128 to 256 times 2, scale context 2048 to 8192 times 4, new KV cache left as a question mark. — The two scaling factors from the question, product left blank.

5. main.py prints a 4.00x relative throughput for GGUF Q4, GPTQ and AWQ, and a teammate concludes the three run equally fast. What is wrong with that conclusion?Analyze

A card, formula rel_tput equals 16 divided by weight_bits, then GGUF Q4 4.00x, GPTQ 4.00x, AWQ 4.00x, all identical, and real GPU speed left as a question mark. — The toy's column from the question. It does not show real GPU speed.

Ship It

This lesson produces outputs/skill-quantization-picker.md. Given hardware, model size, workload type, and quality tolerance, it picks a format and produces a calibration and validation plan.

Author's Note

I'd run main.py before trusting any single memory number in this lesson, including mine. It prints weights, KV, activations, and a GPU fit for BF16, GGUF Q5/Q4, GPTQ, AWQ, FP8, and NVFP4 at four model and load settings, so the KV trap is something you can read off a table instead of taking on faith.

The KV figures in the body come from that script. It assumes 64 layers scaled by model size, 8 KV heads, head dim 128, and FP16 KV, and prints 68.7 GB for a 70B model at 128 concurrent and 2k context. With AWQ weights that totals 107.2 GB, which lands on an H200, not an H100. In the toy, the only 70B row that fits one H100 80GB is NVFP4 with FP8 KV (72.9 GB), in both 70B settings. Real KV depends on the model's actual layer and head counts, so measure yours before you size hardware.

I'd also read the throughput column as a bandwidth argument and nothing more. It is just 16 divided by weight bits, so GGUF Q4, GPTQ, and AWQ all print 4.00x. It says nothing about kernels, which is where the 93, 712, and 741 tok/s figures in the body come from. And the toy has no calibration step at all. Quality is the part you have to measure on your own eval set, and that is what I'd spend the time on before picking a format.

Exercises

Run code/main.py. For a 70B model at 128 concurrent with 2k context, compute the total HBM for each format. Which format lets you fit on one H100 80GB?
You have a 7B coding model. Pick a format and justify. If you were wrong about quality tolerance, what is the recovery path?
Compute the calibration-dataset size needed to calibrate AWQ for a medical domain model. Why is more data not always better?
Read the Marlin-AWQ kernel paper or release notes. Explain in three sentences why AWQ hits 741 tok/s on 7B while raw GPTQ hits ~712.
When does it make sense to combine AWQ weights with FP8 KV cache vs keeping KV at BF16?

Key Terms

Term	What people say	What it actually means
GGUF	"llama.cpp format"	File format bundling K-quant variants; CPU/edge default
Q4_K_M	"Q4 K M"	4-bit K-quant medium; the production GGUF default
GPTQ	"gee pee tee q"	Post-train INT4 with calibration; supports LoRA in vLLM
AWQ	"a w q"	Activation-aware INT4; Marlin kernels; best Pass@1 at INT4
Marlin kernels	"fast INT4 kernels"	Custom CUDA kernels for INT4 weights on GPU; 2.6x (GPTQ) to 10.9x (AWQ) speedup
FP8	"eight-bit float"	Safe precision default on Hopper/Ada/Blackwell
MXFP4 / NVFP4	"microscaling four"	Blackwell 4-bit FP with per-block scale factors
Calibration dataset	"cal data"	Input text used to pick quantization parameters; must match domain
KV cache quantization	"KV INT8"	Separate choice from weights; affects attention accuracy