LearnPython

Inference Metrics — TTFT, TPOT, ITL, Goodput, P99

Four metrics tell you whether an inference deployment works. TTFT is queue plus network plus prefill. TPOT (also called ITL) is the memory-bound decode cost per token. End-to-end latency is TTFT plus TPOT times output length. Throughput is aggregate tokens per second across the fleet. The one that matters for the product is goodput: the fraction of requests that met every SLO constraint at once. High throughput at low goodput means you are producing tokens that reach users too late. Reference numbers for Llama-3.1-8B-Instruct on TRT-LLM in 2026: mean TTFT 162 ms, mean TPOT 7.33 ms, mean E2E 1,093 ms. Report P50, P90, and P99, not just the mean. And watch the measurement trap: GenAI-Perf excludes TTFT from the ITL calculation, LLMPerf includes it, so two tools can disagree on TPOT for the same run.

TIME: ~60 minutesPREREQUISITES: Phase 17 · 04 (vLLM Serving Internals)

Learning Objectives

Define TTFT, TPOT, ITL, E2E, throughput, and goodput precisely and name the component each one measures.
Explain why mean is the wrong statistic for LLM serving and how to read P50/P90/P99.
Construct an SLO multi-constraint (e.g. TTFT<500 ms AND TPOT<15 ms AND E2E<2 s) and compute goodput against it.
Name two benchmark tools that disagree on TPOT for the same run and explain why.

The Problem

"Our throughput is 15,000 tokens per second." So what? If 40% of requests blew past 2 seconds end-to-end, users abandoned the session. Throughput alone does not tell you whether the product works.

Inference has several axes of latency, and each one fails differently. Prefill is compute-bound and scales with prompt length. Decode is memory-bound and scales with batch size. Queuing delay is an operational problem, and network time is a physical-distance problem. Each needs its own metric, each needs percentiles, and you need one composite that answers "did the user get what they expected?" That composite is goodput.

Warm-up0/3

1. In LLM serving, which phase processes the entire prompt in one parallel pass and is compute-bound?Warm-up

Generic setup, no phase names attached. Typed.

2. What mainly limits the speed of a single decode step for one token?Warm-up

Generic mechanism, no bottleneck named. Typed.

3. A chat app streams its reply word by word instead of sending it only when it is complete. What is the main benefit?Warm-up

Two horizontal rows. The streaming row shows six dots spaced out along a line, one word appearing after another. The wait-then-send row shows a single block appearing only near the end. — Everyday setup, typed. No benefit stated, just the two delivery patterns.

Check: The Problem0/1

1. A fleet reports 15,000 tokens per second, yet 40 percent of requests finish after 2 seconds. Why is the throughput number not enough to call the product healthy?Explain

Two boxes: one showing 15,000 tokens per second fleet throughput, the other showing 40 percent of requests finishing after 2 seconds. — The two facts from the stem, typed. No conclusion drawn.

The Concept

TTFT: time to first token

TTFT = queue_time + network_request + prefill_time

Prefill dominates when prompts are long. On Llama-3.3-70B FP8 on H100, a 32k prompt takes ~800 ms of pure prefill. Queue time is scheduler behavior under load. Network request is wire time including TLS. TTFT is the latency the user sees before anything streams back.

TPOT / ITL: inter-token latency

One quantity, many names: TPOT (time per output token), ITL (inter-token latency), decode latency per token. It is the time between consecutive streamed tokens after the first.

TPOT = (decode_forward_time + scheduler_overhead) / tokens_produced

On the same Llama-3.3-70B H100 stack with chunked prefill, mean TPOT is ~7 ms. Without chunked prefill, a long prefill on a neighboring sequence can push TPOT to 50 ms. Watch P99, not the mean.

E2E latency

E2E = TTFT + TPOT * output_tokens + network_response

For long outputs (>500 tokens), TPOT dominates E2E. For short outputs with long prompts, TTFT dominates. Report E2E conditioned on output length.

Throughput

throughput = total_output_tokens / elapsed_time

This is an aggregate number. It tells you how efficiently the fleet runs and nothing about the health of any single request.

Goodput: the metric you actually care about

goodput = fraction of requests meeting (TTFT <= a) AND (TPOT <= b) AND (E2E <= c)

The SLO is a multi-constraint. A request counts as good only if every constraint held, and goodput is the share of requests that did. High throughput at 60% goodput is a failing deployment. Lower throughput at 99% goodput is the target.

In 2026, goodput is the metric used in MLPerf Inference v6.0 submissions and in internal SLA tracking at AI platform providers.

Why the mean misleads

LLM latency distributions are right-skewed. Say a decode batch has one long-prefill neighbor: it can ship 500 tokens at ~7 ms TPOT and 20 tokens at ~60 ms. Mean TPOT comes out at 9 ms while P99 TPOT is 65 ms. Users hit that P99 regularly, and that is when they leave.

Report the triple (P50, P90, P99). For user experience, P99 is the one to optimize.

Reference numbers: Llama-3.1-8B-Instruct on TRT-LLM, 2026

mean TTFT: 162 ms
mean TPOT: 7.33 ms
mean E2E: 1,093 ms
P99 TPOT: varies from 10 to 25 ms depending on chunked-prefill configuration.

These are the published NVIDIA reference points. They shift with model size (70B would show 3-5x), hardware (H100 vs B200 ~3x), and load.

The measurement trap

Two of the most-used 2026 benchmark tools disagree on TPOT for the same run:

NVIDIA GenAI-Perf: excludes TTFT from the ITL calculation. ITL starts from token 2.
LLMPerf: includes TTFT. ITL starts from token 1.

Take a request with TTFT 500 ms and 100 output tokens in 700 ms of total decode. GenAI-Perf reports ITL = 700/99 = 7.07 ms. LLMPerf reports ITL = 1200/100 = 12.00 ms. Same run, different number, depending on the tool.

State which tool you used and publish its definition.

Constructing an SLO

A reasonable consumer-facing SLO for a 70B chat model in 2026:

TTFT P99 <= 800 ms.
TPOT P99 <= 25 ms.
E2E P99 <= 3 s for <300-token outputs.
Goodput target >= 99%.

Enterprise SLOs tighten TTFT (200-400 ms) and loosen E2E. What matters is that you write the bounds down, measure all three, and track goodput as one composite.

How to measure

Run real traffic or a realistic synthetic load (LLMPerf with --mean-input-tokens 800 --stddev-input-tokens 300 --mean-output-tokens 150).
Target 2x peak concurrency for the benchmark run.
Run 30-50 iterations and take percentiles of the combined sample.
Publish the tool name, tool version, model, hardware, concurrency, and prompt distribution.

Check: The Concept0/24

1. Which components make up TTFT (time to first token)?Define

A single horizontal span running from a dot labelled request sent to a dot labelled first token arrives, bracketed underneath and marked TTFT, with no internal breakdown drawn. — TTFT drawn as one undivided interval. Typed.

2. What does TPOT measure in LLM serving?Define

Six dots on a line representing streamed tokens, the first drawn in a different color. A bracket spans the gap between the third and fourth dots, labelled this gap equals question mark. — Concrete instance of the quantity, typed. Not every gap is marked.

3. How does ITL relate to TPOT in most benchmark tools?Define

Two boxes, one labelled TPOT and one labelled ITL, with a small dashed box between them holding an equals-question-mark symbol. — Setup only, typed. The relationship is left unresolved.

4. What does end-to-end (E2E) latency add up to?Define

A single horizontal span running from a dot labelled request sent to a dot labelled last token streamed, bracketed and marked E2E, with no internal breakdown drawn. — E2E drawn as one undivided interval. Typed.

5. What does throughput tell you in LLM serving?Define

Three request rows each labelled streaming tokens, with arrows converging into a dashed box holding a question mark labelled one number. — Setup only, typed. What the combined number measures is not resolved.

6. Which metric matters most for the product, and why?Define

A table of three SLO profiles from a real run of code/main.py with their goodput values: loose 100%, target 75.95%, tight 42.15%. — Real values from a run of code/main.py with its default settings.

7. What does P99 latency represent?Define

A terminal-style box showing one real printed line from code/main.py listing TPOT percentiles P50, P90 and P99 with their millisecond values. — Real printed line from one run of code/main.py; other lines omitted.

8. How is a request judged against an SLO multi-constraint?Define

Placeholder names from the lesson's own goodput formula. Typed.

9. What is the 'tool trap' in LLM benchmarking?Define

Two boxes, GenAI-Perf and LLMPerf, with a dashed question-mark box between them and a caption saying the same trace gives a different reported ITL in each tool. — Setup only, typed. The cause is left unresolved.

10. A user's prompt grows from 2,000 to 32,000 tokens while the output length stays the same. Which metric moves most, and why?Explain

Two boxes for request 1 with a 2,000-token prompt and request 2 with a 32,000-token prompt, each marked same output length, with no latency numbers shown. — Setup only, typed. No metric values are drawn.

11. Why is the mean a poor statistic for LLM latency?Explain

A bar chart of 30 per-token decode times in one batch: most bars are short, around 7 milliseconds, and a few near the right are much taller, around 60 milliseconds. — Illustrative: exaggerated for visibility, not to scale. The lesson's own example is 500 tokens at about 7 ms and 20 at about 60 ms, a 25 to 1 ratio.

12. For outputs longer than 500 tokens, which term dominates end-to-end latency?Explain

A single card reading an output of more than 500 tokens, with a caption about which term in E2E grows most as output length grows. — The condition given in the stem, typed.

13. Why does goodput combine the TTFT, TPOT and E2E bounds with AND instead of reporting three separate pass rates?Explain

Setup only, typed. No percentages are filled in.

14. Which set is a reasonable consumer-facing SLO for a 70B chat model in 2026?Explain

Template structure from the lesson's Constructing an SLO section, values left blank.

15. Which benchmark methodology does the measurement recipe recommend?Explain

A three-box pipeline: load generator, target service, and a dashed results box, with two blanks below reading concurrency: question mark times peak, and iterations: question mark. — Pipeline structure, typed. Concurrency, iteration count and how results are reported are all left blank.

16. On the 8B TRT-LLM reference run, mean TTFT is 162 ms, mean TPOT is 7.33 ms and mean E2E is 1,093 ms. Ignoring response-side network time, about how many output tokens does an average request produce?Apply

A table of the three given means: TTFT 162 ms, TPOT 7.33 ms, E2E 1093 ms. — The three numbers given directly in the question.

17. A request has TTFT 500 ms and 100 output tokens finishing decode in 700 ms. What ITL do GenAI-Perf and LLMPerf report?Apply

A table of the given numbers for one request: TTFT 500 milliseconds, 100 output tokens, and 700 milliseconds of total decode time. — The numbers given directly in the question.

18. Four requests, as (TTFT ms, TPOT ms, E2E ms): A (300, 8, 1500), B (600, 7, 1900), C (200, 12, 2400), D (450, 14, 1800). The SLO is TTFT <= 500, TPOT <= 15, E2E <= 2000. What is the goodput?Apply

A table of four requests A through D with their TTFT, TPOT and E2E values in milliseconds, and a caption giving the SLO: TTFT 500 or less, TPOT 15 or less, E2E 2000 or less. — The four requests' values, exactly as given in the question.

19. Your dashboard shows mean TPOT at 7 ms and throughput up 20% after a config change, but users report stalls. P99 TPOT is 65 ms. What should you do first?Analyze

A dashboard card listing mean TPOT 7 milliseconds, throughput up 20 percent, P99 TPOT 65 milliseconds, and user reports of stalls. — The dashboard values given directly in the question.

20. Team A reports a TPOT of 7 ms from GenAI-Perf. Team B reports 12 ms from LLMPerf and concludes its own engine is slower. What is wrong with that conclusion?Analyze

Two cards: Team A using GenAI-Perf reports TPOT 7 milliseconds, Team B using LLMPerf reports TPOT 12 milliseconds, with a note that Team B concludes its own engine is slower. — The two teams' tools, numbers and conclusion, exactly as given in the question.

21. Users complain about slow starts only on prompts of about 32,000 tokens, while P99 TPOT looks fine. Which component should you suspect first?Analyze

A card listing the symptom: complaints of slow starts, at a prompt size around 32,000 tokens, while P99 TPOT looks fine. — The symptom pattern given directly in the question.

22. You must gate deploys in CI for a 70B chat service. Which gate design follows the lesson?Design

A single card reading gate deploys in CI for a 70B chat service, with a caption noting that four candidate gate designs are offered as options. — The task given directly in the question.

23. An enterprise customer wants answers to start almost immediately and accepts long total times for big outputs. How should you adjust the SLO from the consumer profile?Design

A card listing what the enterprise customer wants: first token almost immediately, and a long total time accepted for big outputs. — The customer's two preferences, exactly as given in the question.

24. You publish benchmark results that others will compare with their own. Which report is complete enough to compare?Design

Setup only, typed. No report contents are listed.

Use It

code/main.py is a toy goodput calculator in stdlib Python. It simulates 2,000 requests with right-skewed latency (exponential queue time, prefill proportional to prompt length, noisy per-token decode with occasional tail spikes), applies three SLO profiles, and prints goodput for each. It also computes TPOT both ways, GenAI-Perf style and LLMPerf style, on the same trace.

Two simplifications to keep in mind. TTFT in the toy is queue plus prefill, with no network term. And the "KEY FINDING" block at the bottom is fixed text, not computed from the run, so trust the numbers printed above it.

Check: Use It0/6

1. In the Use It toy run, LLMPerf-style mean TPOT comes out about 1 ms above GenAI-Perf style on the same trace. What puts it higher?Explain

A table comparing mean TPOT on one trace: GenAI-Perf style at 7.69 milliseconds, LLMPerf style at 8.71 milliseconds, with a note that the gap is about 1 millisecond. — Real values from a run of code/main.py, both TPOT definitions computed on the same trace.

2. In the toy goodput calculator, which output should you not treat as a computed result?Explain

A terminal-style box with the real printed output of one run of code/main.py: a TTFT percentile line, a TPOT percentile line, the tool-trap line, three goodput lines with their tags, and the KEY FINDING block. — Real output of one run of code/main.py, condensed to fit; banner lines removed.

3. In main.py, a request has queue_ms of 40 and a 2,048-token prompt, and prefill_ms is prompt_len * 0.05. What does ttft_ms return?Apply

A table of the given inputs: queue_ms 40, prompt length 2,048 tokens, and prefill_ms defined as prompt_len times 0.05, with a note that these are the fields ttft_ms has available to it. — The values and formula given directly in the question.

4. In main.py, a request has output_tokens equal to 1. What does tpot_genaiperf() return?Apply

A code excerpt of tpot_genaiperf with an if statement for output_tokens less than or equal to 1, whose return value is hidden behind three question marks, followed by the normal-case return line. — Condensed from code/main.py; the guarded branch's return value is hidden.

5. main.py computes a goodput of 97 percent for one SLO profile. Which tag does it print?Apply

A code excerpt of the tag expression: SHIPPABLE if g is at least 0.99, otherwise a hidden value if g is at least 0.95, otherwise FAILING, with a note that this run has g equal to 0.97. — Condensed from code/main.py; the middle tag's name is hidden.

6. In the toy, tightening only the TPOT bound barely changes goodput, and P99 TPOT stays near 9 ms despite tail spikes. What explains this?Analyze

A card listing what is observed: tightening only the TPOT bound, goodput barely changes, and P99 TPOT stays near 9 milliseconds. — The observation given directly in the question.

Ship It

This lesson produces outputs/skill-slo-goodput-gate.md. Given a workload and SLO, it produces a CI/CD-ready benchmark recipe that gates deploys on goodput rather than throughput.

Check: Ship It0/1

1. What does the slo-goodput-gate skill gate deploys on?Explain

Setup only, typed. The gate condition is not shown.

Author's Note

I'd run main.py first and read the three goodput lines before the prose. With the defaults (2,000 requests, seed 7, 2% tail spikes) the loose SLO passes every request, the target SLO (TTFT<500, TPOT<15, E2E<2000) lands around 76%, and the tight one around 42%. That is the multi-constraint idea in one screen.

The part that surprised me is which constraint does the damage. Per-request TPOT in the toy is an average over 50 to 300 decode tokens, so the tail spikes get smoothed out and P99 TPOT stays near 9 ms. The goodput loss comes from E2E and TTFT. A spike that hurts the user can vanish from a per-request TPOT number, which is a good argument for looking at percentiles of E2E alongside it.

I'd also read the tool-trap line as a reminder to cite the tool. The two TPOT definitions differ by about 1 ms on the mean here, because the LLMPerf version folds queue and prefill into the numerator. If you try the first exercise, expect tightening TPOT alone to move goodput very little in this toy. Pooling per-token latencies instead of averaging per request would be the change to try next.

Exercises

Run code/main.py. Generate a distribution with 1% tail spike. How does goodput change when you tighten P99 TPOT from 30 ms to 15 ms?
A vendor quotes "15,000 tok/s on Llama 3.3 70B H100". Name three questions to ask before trusting it.
Why does chunked prefill protect P99 TPOT but not mean TPOT?
Construct a consumer SLO for a voice assistant (first token is heard, not read). Which metric is most user-visible?
Read the LLMPerf README and the GenAI-Perf docs. Identify three other metrics where the tools disagree.

Key Terms

Term	What people say	What it actually means
TTFT	"time to first token"	Queue + network + prefill; dominated by prefill at long prompts
TPOT	"time per output token"	Memory-bound decode cost per token after first
ITL	"inter-token latency"	Same as TPOT in most tools (not all, see GenAI-Perf)
E2E	"end to end"	TTFT + TPOT * output_len; response-side network on top
Throughput	"tok/s"	Fleet efficiency; useless without latency percentiles
Goodput	"SLO-met rate"	Fraction of requests meeting every SLO constraint simultaneously
P99	"tail"	1-in-100 worst-case latency; the user experience metric
SLO multi-constraint	"the joint"	AND of all three latency bounds; a request fails if any one is violated
GenAI-Perf vs LLMPerf	"the tool trap"	Tools disagree on whether ITL includes TTFT