TAI BUI
Back to curriculum
BuildPython

Load Testing LLM APIs — Why k6 and Locust Lie

Traditional load testers were not built for streaming responses, variable output lengths, token-level metrics, or GPU saturation. Two traps catch most teams. The GIL trap: Locust's token-level measurement runs tokenization under the Python GIL, where it competes with request generation under heavy concurrency. The tokenization backlog inflates reported inter-token latency, so the client is the bottleneck, not the server. The prompt-uniformity trap: identical prompts in a loop test one point on the token distribution, but real traffic has variable length and diverse prefix matches. LLMPerf handles this with `--mean-input-tokens` and `--stddev-input-tokens`. Tool mapping in 2026: LLM-specialized tools (GenAI-Perf, LLMPerf, LLM-Locust, guidellm) for token-level accuracy; **k6 v2026.1.0** with **k6 Operator 1.0 GA (Sept 2025)** for streaming-aware, Kubernetes-native distributed runs via TestRun/PrivateLoadZone CRDs, best for CI/CD gates; Vegeta for Go constant-rate saturation; Locust 2.43.3 only with the LLM-Locust extension for streaming. Load patterns: steady-state, ramp, spike (autoscaling test), soak (memory leaks).

TIME: ~75 minutesPREREQUISITES: Phase 17 · 08 (Inference Metrics), Phase 17 · 03 (GPU Autoscaling)

Learning Objectives

  • Explain the two anti-patterns (GIL trap, prompt-uniformity trap) that make generic load testers lie for LLM APIs.
  • Pick a tool for a given purpose: LLMPerf (benchmark run), k6 + streaming extension (CI gate), guidellm (large-scale synthetic), GenAI-Perf (NVIDIA reference).
  • Design four load patterns (steady, ramp, spike, soak) and name the failure mode each catches.
  • Build a realistic prompt distribution using mean + stddev of input tokens rather than fixed length.

The Problem

Say you k6-test your LLM endpoint at 500 concurrent users and it holds. You ship. In production, at 200 actual users, the service falls over: P99 TTFT explodes and the GPUs pin.

Two things went wrong. First, k6 sent 500 identical prompts, so request coalescing and prefix caching made it look like you were handling 500 concurrent decodes when you were actually handling one. Second, k6 does not track inter-token latency on streaming responses the way a reader experiences it. It sees one HTTP connection, not 500 tokens arriving at varying intervals.

Load testing for LLMs is its own discipline.

Warm-up0/3
1. What does TTFT (time to first token) measure?Warm-up
A timeline from a dot labelled request sent to a dot marking the end of an unlabeled span A, followed by five more dots spaced out as span B, tokens arriving one by one. A bracket under span A reads span A equals question mark.
Generic timeline, typed. Which span is TTFT is left open.
2. Why do SLAs usually gate on P99 latency instead of the median?Warm-up
A hundred sorted latency bars, almost all short and a handful on the far right much taller, with notes that the median sits in the short crowd while a few requests run far longer. A line asks why gate on the slow end instead of the median.
Generic sorted-latency setup, typed. Why to gate on the tail is left open.
3. A traffic spike arrives and the autoscaler adds GPU replicas. Why does latency still suffer for a while?Warm-up
A bar chart of request rate before and during a spike, with an arrow down to a new replica box marked provisioning, and a note that requests keep arriving while it is not ready. A line asks why latency still suffers for a while.
Generic autoscaling setup, typed. Why latency still suffers is left open.
Check: The Problem0/1
1. A k6 test at 500 concurrent users passes, but production falls over at 200 real users. Looking at the test, what is the most likely cause?Analyze
A table contrasting the k6 test, 500 concurrent users with the same prompt each time, against production, 200 real users where TTFT spikes and GPUs pin, with a note that the question asks what in the test explains the gap.
Setup: the test and the production incident, exactly as given in the question.

The Concept

The GIL trap (Locust)

Locust is Python, and it runs tokenization client-side under the GIL. Under high concurrency the tokenizer queues behind request generation, and the reported inter-token latency includes that backlog. You think the server is slow when it's the test harness.

The fix is the LLM-Locust extension, which moves tokenization to separate processes, or a compiled-language harness (k6, or LLMPerf using tokenizers.rs).

The prompt-uniformity trap

Most load testers make it easy to configure a single prompt. In a loop test of 10,000 iterations, that exact prompt goes out each time. The server sees the same prefix on every request, prefix cache hits approach 100%, and throughput looks great.

The fix is to sample from a prompt distribution. LLMPerf takes --mean-input-tokens 500 --stddev-input-tokens 150, which gives diverse lengths and diverse content.

Four load patterns

  1. Steady-state: constant RPS for 30-60 min. Catches baseline performance regressions.
  2. Ramp: linearly increase RPS from 0 to target over 15 min. Catches the capacity breakpoint and warm-up anomalies.
  3. Spike: a sudden 3-10x RPS for 2 min, then back. Catches autoscaling latency, queue saturation, and cold-start impact.
  4. Soak: steady-state for 4-8 hours. Catches memory leaks, connection-pool drift, and observability overflow.

2026 tool mapping

LLMPerf (Anyscale) is Python with Rust-backed tokenization. It samples mean/stddev prompts and is streaming-aware. It's the best default for performance runs.

NVIDIA GenAI-Perf is NVIDIA's reference harness. It uses the Triton client and covers a wide range of metrics. Its ITL excludes TTFT while LLMPerf's includes it, so the two tools produce different TPOT for the same server.

LLM-Locust (TrueFoundry) is a Locust extension that fixes the GIL trap. You keep the familiar Locust DSL and get streaming metrics.

guidellm does large-scale synthetic benchmarking.

k6 v2026.1.0 + k6 Operator 1.0 GA (Sept 2025):

  • k6 itself (Go, compiled, no GIL) added streaming-aware metrics.
  • The k6 Operator uses TestRun / PrivateLoadZone CRDs for Kubernetes-native distributed testing.
  • Best for CI/CD gates and SLA testing.

Vegeta is Go and simpler than k6. It does constant-rate HTTP saturation. It isn't LLM-aware, but it works for gateway and rate-limit testing.

Locust 2.43.3 stock has the GIL trap for LLM work, so use it only with the LLM-Locust extension.

SLA gate in CI

Run k6 on the PR with:

  • 30-50 iterations at baseline RPS.
  • Gate: P50/P95 TTFT, 5xx under 5%, TPOT under threshold.
  • Break the build on breach.

Realistic prompt distribution

Build it from real traffic samples if you have them, or from published distributions (ShareGPT prompts for chat, HumanEval for code). Feed the mean and stddev to LLMPerf. Skip the loop with one prompt.

Numbers you should remember

  • k6 Operator 1.0 GA: September 2025.
  • k6 v2026.1.0: streaming-aware metrics.
  • Typical LLMPerf run: 100-1000 requests at concurrency X.
  • Typical CI gate: 30-50 iterations per PR.
  • Four patterns: steady, ramp, spike, soak.
Check: The Concept0/26
1. What is LLMPerf?Define
A card naming LLMPerf among the other 2026 tools this lesson names: GenAI-Perf, LLM-Locust, guidellm, k6 Operator and Vegeta. A line asks what it is.
Setup: the name, among its sibling tools. What it is is left open.
2. What is GenAI-Perf?Define
A card naming GenAI-Perf among the other 2026 tools this lesson names: LLMPerf, LLM-Locust, guidellm, k6 Operator and Vegeta. A line asks what it is.
Setup: the name, among its sibling tools. What it is is left open.
3. What is LLM-Locust?Define
A card naming LLM-Locust as something built on top of stock Locust 2.43.3, which the lesson says has a trap for LLM work. A line asks what LLM-Locust is.
Setup: the name and that it builds on stock Locust. What it actually is is left open.
4. What is guidellm?Define
A card naming guidellm among the other 2026 tools this lesson names: LLMPerf, GenAI-Perf, LLM-Locust and k6 Operator. A line asks what it is.
Setup: the name, among its sibling tools. What it is is left open.
5. What is the k6 Operator?Define
A card naming the k6 Operator as a separate piece alongside k6 itself, version 2026.1.0, in the lesson's tool mapping. A line asks what it is.
Setup: the name, placed next to k6 itself. What it actually is is left open.
6. What is a steady-state load pattern?Define
A card naming the steady-state pattern among its three siblings: ramp, spike and soak. A line asks what it is.
Setup: the name, among its sibling patterns. What it is is left open.
7. What is a ramp load pattern?Define
A card naming the ramp pattern among its three siblings: steady-state, spike and soak. A line asks what it is.
Setup: the name, among its sibling patterns. What it is is left open.
8. What is a spike load pattern?Define
A card naming the spike pattern among its three siblings: steady-state, ramp and soak. A line asks what it is.
Setup: the name, among its sibling patterns. What it is is left open.
9. What is the GIL trap in Locust-based LLM load testing?Define
A card naming the GIL trap as one of two traps this lesson names, the other being the prompt-uniformity trap. A line asks what it is.
Setup: the name, next to its sibling trap. The mechanism is left open.
10. What is the prompt-uniformity trap?Define
A card naming the prompt-uniformity trap as the second of the two named traps, the other being the GIL trap. A line asks what it is.
Setup: the name, next to its sibling trap. The mechanism is left open.
11. Which failure mode does the soak pattern catch?Define
A card naming the soak pattern alongside its three siblings: steady-state, ramp and spike. A line asks which failure mode it catches.
Setup: the name, among its sibling patterns. Which failure mode it catches is left open.
12. Why does the lesson suggest a compiled-language harness such as k6 for LLM load tests?Explain
Two harness boxes doing the same client-side tokenization work, one Python and one compiled, with a note that the lesson recommends one of them for LLM load tests and a line asking why.
Setup: the two harnesses and their shared client-side task, typed. Why one is preferred is left open.
13. How should you build a realistic prompt distribution for a load test?Explain
A single box labelled one prompt, the same text every iteration, with a note that the question asks for a better way to build the set of test prompts. A line asks what a more realistic approach looks like.
Setup: the current one-prompt source, typed. The better approach is left open.
14. Why does sampling input length from a mean and stddev give a more honest test than a fixed length?Explain
Two prompt-length policies side by side: fixed length, every request the same, and sampled length, drawn from a given mean and stddev. A line asks why the sampled version makes for a more honest test.
Setup: the two policies, with sampling already named as the method. Why it is more honest is left open.
15. Which pairing of load pattern and failure mode it catches is correct?Explain
Two unconnected columns, one with the names steady-state, ramp and spike, the other with three placeholder boxes reading finds X, finds Y and finds Z, with no lines drawn between them. A line asks which pairing of pattern and failure mode is correct.
Setup: the three pattern names and three blank failure-mode slots, unpaired. Typed.
16. Why does a spike test expose autoscaling latency and cold starts?Explain
A bar chart jumping from an illustrative baseline to an illustrative spike request rate, next to a box labelled new replica, still starting up, with a note that the question asks why a spike test specifically surfaces this. A line asks why.
Setup: illustrative request-rate numbers and the not-yet-ready replica, typed. Why a spike surfaces this is left open.
17. What does a k6 CI SLA gate on a pull request look like?Explain
A pull request box with an arrow into a dashed box labelled k6 run, question mark, with a note that the question asks what that run actually checks and how.
Setup: the PR-to-gate flow, typed. What the gate checks is left open.
18. Which 2026 tool combination is positioned as best for CI/CD SLA gates and Kubernetes-native distributed runs?Explain
Two requirement boxes, a CI/CD SLA gate that fails the build on breach, and Kubernetes-native distributed test runs, with a note that four candidate tools are offered to satisfy both. No tool names are shown.
Setup: the two requirements, typed. Which tool or combination satisfies both is left open.
19. Why can GenAI-Perf and LLMPerf report different TPOT numbers for the same server?Explain
Two boxes, GenAI-Perf and LLMPerf, each reporting its own TPOT for the same server, with a note that both measure the very same requests and a line asking why the two numbers can differ.
Setup: the two tools and that they share a server, typed. Why their TPOT differs is left open.
20. Your baseline is 10 RPS and you run a spike test at the lesson's 3-10x multiplier. What request rate do you hit during the spike?Apply
A table of the two given numbers: baseline RPS of 10, and a spike multiplier of 3x to 10x, with a note that the question asks for the request rate during the spike.
The two numbers given directly in the question.
21. A service passes a 45-minute steady-state test and a 2-minute spike, then degrades after 6 hours in production with memory climbing. Which test would have caught it?Analyze
A table of three events: a 45-minute steady-state test that passed, a 2-minute spike test that passed, and 6 hours in production where memory is climbing. A note says the question asks which test design would have caught this.
Setup: the test history and the production symptom, exactly as given in the question.
22. A Locust run shows inter-token latency far above what the server's own metrics report. The harness tokenizes on the client at high concurrency. Which diagnosis fits?Analyze
A table contrasting the inter-token latency Locust reports, far above, against the server's own metrics, normal, with a note that the harness tokenizes on the client at high concurrency. A line asks which diagnosis fits.
Setup: the mismatch and the client-side tokenization detail, exactly as given in the question.
23. A CI gate loops one fixed prompt 40 times per PR and passes every time, yet production P99 TTFT keeps getting worse. What is wrong with the gate?Analyze
A table of two rows: the CI gate, one fixed prompt looped 40 times, passing every time, and production, where P99 TTFT keeps getting worse. A note says the question asks what is wrong with the gate.
Setup: the gate's design and the production trend, exactly as given in the question.
24. You must add a pull-request check that fails the build when LLM tail latency regresses. Which design fits?Design
A single card reading a PR check that fails the build on tail-latency regression, with a note that four candidate designs are offered as options.
The task given directly in the question.
25. A launch will multiply traffic suddenly, and you only have samples of real prompts. Which plan best checks that autoscaling copes?Design
A card listing what is known before launch: traffic will multiply suddenly, real prompt samples are available, and how autoscaling copes is not yet known. A note says four candidate test plans are offered.
The facts given directly in the question.
26. Your team already writes Locust scripts and wants LLM streaming metrics without a rewrite or the GIL trap. Which choice fits?Design
A card listing what this team already has and wants: existing Locust scripts, a wish for LLM streaming metrics, and no rewrite or GIL trap. A note says four candidate choices are offered.
The facts given directly in the question.

Use It

code/main.py simulates a load test with a uniform workload and a realistic prompt distribution, reports TTFT P50, P99, mean and cache hits for each, and demonstrates the uniform-prompt trap. It does not model TPOT.

Check: Use It0/5
1. In main.py the uniform workload is 500 requests that all share the prefix "single_prefix". How many cache hits does simulate() report?Apply
A box describing 500 requests that all carry the prefix single_prefix, with a note that the cache starts empty before the first request. A line asks how many of the 500 report a cache hit.
Setup: the uniform workload as make_uniform_workload builds it in main.py, typed.
2. For the uniform workload of 500 requests (1 miss at 800 ms, 499 hits at 80 ms), what P99 TTFT does simulate() return?Apply
A table of the given samples: 1 request at 800 milliseconds and 499 requests at 80 milliseconds. A note says the question asks for the P99 of these 500 sorted samples.
The sample counts given directly in the question.
3. Four requests each carry a different prefix_hash and the cache starts empty. Using main.py's constants, what is the mean TTFT?Apply
A table of four requests, each with a different prefix_hash marked unique, with a note that the cache starts empty and every prefix is new to it. A line asks for the mean TTFT.
Setup: the four requests' distinct prefixes, as given in the question, typed.
4. main.py runs the same 500-request realistic workload at concurrency 10, 50 and 200. How do the printed rows compare?Apply
Three boxes for concurrency 10, 50 and 200, each run on the same 500 requests, with a note that the printed rows from all three runs are being compared and a line asking how.
Setup: the same workload run at three concurrency settings, typed.
5. In the Author's Note run of main.py, uniform and realistic workloads both show a P50 TTFT of 80 ms. What should that tell you about gating?Apply
Two workload boxes, uniform and realistic, both reporting a P50 TTFT of 80 milliseconds, with a note that the question asks what this equal median tells you about gating.
The P50 fact given directly in the Author's Note, typed.

Ship It

This lesson produces outputs/skill-load-test-plan.md. Given a workload and an SLA, it picks a tool and designs the four load patterns.

Author's Note

I'd run main.py before trusting any of the prose. It builds two 500-request workloads, one where every request carries the same 2000-token prompt and prefix, and one with prompt lengths drawn from a Gaussian around 500 tokens spread over 80 prefixes. The uniform run gets 499 cache hits and a P99 TTFT of 80 ms. The realistic run gets 421 hits and a P99 of 800 ms.

Read the numbers with the toy's limits in mind. TTFT is a constant: 80 ms on a hit, 800 ms on a miss. Prompt length never enters it, and TPOT_MS is defined but unused, so the script shows the cache effect and nothing about token-level latency. It also gives identical output at concurrency 10, 50, and 200, because the cache is shared across the whole run and the concurrency value only groups requests. That means the script says nothing about how concurrency changes the tail.

The gap is still the point. A P50 of 80 ms looks the same for both workloads. The P99 (80 ms vs 800 ms) is what separates them, so I'd gate on the tail and not the median. For a real run, I'd take prompt shapes from production traffic if I had them and fall back to LLMPerf's mean and stddev flags if I didn't.

Exercises

  1. Run code/main.py. Compare uniform vs realistic distribution. Where is the gap?
  2. Write the k6 script for a CI gate: TTFT P95 < 800 ms at 100 concurrent, runtime 5 minutes.
  3. Your soak test shows memory growing 50 MB/hour. Name three causes and the instrumentation to pick between them.
  4. Spike test from 10 RPS to 100 RPS. What's the expected recovery time if Karpenter + vLLM production-stack are in place (Phase 17 · 03 + 18)?
  5. GenAI-Perf reports TPOT=6ms; LLMPerf reports TPOT=11ms on the same server. Explain.

Key Terms

TermWhat people sayWhat it actually means
LLMPerf"the LLM harness"Anyscale benchmark tool, streaming-aware
GenAI-Perf"NVIDIA tool"NVIDIA reference harness
LLM-Locust"Locust for LLMs"Locust extension fixing GIL trap
guidellm"synthetic benchmark"Large-scale synthetic tool
k6 Operator"K8s k6"CRD-based distributed k6
GIL trap"Python client overhead"Tokenization backlog inflates reported latency
Prompt-uniformity trap"single-prompt lie"Loop with same prompt hits cache, inflates throughput
Steady-state"constant load"Flat RPS for N minutes
Ramp"linear up"0 to target over duration
Spike"burst test"Sudden multiplier then revert
Soak"long test"Hours for leak detection

Further Reading

Code

Ships With