TAI BUI
Back to curriculum
LearnPython

Chaos Engineering for LLM Production

Chaos engineering for LLM services only works once you have SLIs and SLOs, observability, automated rollback, runbooks, and someone on call. With those in place, four planes (control, target, safety, observability) plus a feedback loop into your SLOs keep experiments contained. The safety plane pauses an experiment when daily error-budget burn passes 2x expected. LLM stacks add their own experiments: memory overload, network failures, provider outages, malformed prompts, and KV cache eviction storms. Run a small canary weekly, a game day monthly, and a cross-team audit quarterly.

TIME: ~60 minutesPREREQUISITES: Phase 17 · 23 (SRE for AI), Phase 17 · 13 (Observability)

Learning Objectives

  • Name the five chaos engineering prerequisites (SLI/SLO, observability, rollback, runbooks, on-call) and explain why skipping any breaks the practice.
  • Diagram the four planes (control, target, safety, observability) and the feedback loop into SLO.
  • Enumerate five LLM-specific experiments (memory overload, network fail, provider outage, malformed prompt, KV eviction storm).
  • Pick a tool (Harness, LitmusChaos, or Chaos Mesh) given your stack.

The Problem

Chaos testing in traditional stacks is well established. LLM stacks add failure modes that older playbooks never covered. A 4K-token prompt with a poison character stalls the tokenizer for 12 seconds. An upstream provider returns 429s, your gateway retries, and the retries amplify concurrency until the service runs out of memory. A burst of load causes a KV cache eviction storm, and the re-prefill cascade that follows saturates compute.

Unit tests catch none of these. Chaos engineering is how you find them before your users do.

Warm-up0/3
1. Under a 99.9% availability SLO, what is the error budget?Warm-up
A card stating a 99.9% success target over one period, a day or a month, with the fraction of requests allowed to fail inside that period left as a question mark.
The target exactly as given in the question. No fraction is computed.
2. Which observability signal lets you follow one request across several services?Warm-up
A card stating the goal of following one request across several services, with several different kinds of signals available, and which signal does this and why left as a question mark.
Generic observability setup, typed. No signal type is named.
3. A service starts failing under load, and every client immediately retries each failed call. What usually happens next?Warm-up
A card stating that a service starts failing under load and every client immediately retries each failed call, with what usually happens next left as a question mark.
The setup exactly as given in the question. No outcome is shown.

The Concept

Prerequisites

Don't run chaos in production without these five things:

  1. SLI/SLO. Defined service-level indicators and objectives. Without them you can't say whether an experiment hurt.
  2. Observability. Traces, metrics, and logs, wired to dashboards.
  3. Automated rollback. The policy-flag rollback from Phase 17 · 20.
  4. Runbooks. Structured ones, as in Phase 17 · 23.
  5. On-call. Someone has to respond.

Miss any one and the chaos becomes a real incident.

Four planes plus a feedback loop

Control plane. The experiment scheduler: a Litmus workflow, a Chaos Mesh schedule, or the Harness UI.

Target plane. The things you break: services, pods, nodes, load balancers, data stores.

Safety plane. Kill switch, suppression windows, blast-radius limits, and error-budget gates.

Observability plane. Your normal metrics, plus trace-ID correlation so you can tell chaos-induced failures from natural ones.

Feedback loop. Findings flow back into SLO adjustments, runbook updates, and code fixes.

Guardrails are mandatory

  • Burn-rate alert: pause the experiment if daily error-budget burn exceeds 2x expected.
  • Suppression windows: silence non-experiment alerts inside the blast radius while the experiment runs.
  • Trace-ID correlation: tag every experiment-induced error so on-call can dedupe.

Five LLM-specific experiments

  1. Memory overload. Force a KV cache preemption storm with long-context requests at high concurrency. Observe whether the service sheds load gracefully or crashes.

  2. Network failure. Cut connectivity between the inference gateway and the provider. Observe whether fallback kicks in within SLA (Phase 17 · 19).

  3. Provider outage simulation. Return 429 on 100% of OpenAI calls. Observe whether routing fails over to Anthropic (Phase 17 · 16, 19).

  4. Malformed prompt. Inject a tokenizer-stalling payload, such as deeply nested unicode or a huge UTF-8 codepoint. Observe whether a single request can lock up a worker.

  5. KV eviction storm. Saturate the vLLM block budget to force eviction. Observe whether LMCache recovers or the service degrades.

Cadence

  • Weekly: small canary experiments in staging, maybe 5% of prod.
  • Monthly: a scheduled game day on one specific scenario, with cross-team attendance and a postmortem.
  • Quarterly: a cross-team resilience audit and a dependency map update.

Tooling

  • Harness Chaos Engineering is commercial. It offers AI-derived experiment recommendations, blast-radius downscaling, and MCP tool integration.
  • LitmusChaos is CNCF graduated and Kubernetes workflow-based.
  • Chaos Mesh is CNCF sandbox and Kubernetes-native, with a CRD style.
  • Gremlin is commercial with broad support.
  • AWS FIS and Azure Chaos Studio are the managed cloud offerings.

Starting small

Your first experiment: kill one decode replica under steady traffic and watch the rerouting and recovery. If that works and looks safe, graduate to network chaos.

Your first LLM-specific experiment: inject a provider 429 for 5 minutes and watch the fallback. Most teams find their fallback wasn't fully tested.

Numbers you should remember

  • Four planes: control, target, safety, observability.
  • Burn-rate pause: 2x expected daily budget burn.
  • Cadence: weekly canary, monthly game day, quarterly audit.
  • Five LLM experiments: memory, network, provider, malformed prompt, KV storm.
Check: The Concept0/23
1. Which five things must exist before you run chaos in production?Define
A card naming the Prerequisites heading, tagged to The Concept section, with what the five things are left as a question mark.
The heading as the lesson names it. What the five things are is the question.
2. What is an SLI/SLO pair in the context of chaos engineering?Define
A card naming SLI / SLO, tagged to the Key Terms section, with what it is left as a question mark.
The term as the lesson names it. What it is is the question.
3. What does blast radius mean for a chaos experiment?Define
A card naming blast radius, tagged to the Key Terms section, with what it means left as a question mark.
The term as the lesson names it. What it means is the question.
4. Which set names the four planes plus the feedback loop in the chaos architecture?Define
A card naming the Four planes plus a feedback loop heading, tagged to The Concept section, with what they are and where the loop leads left as a question mark.
The heading as the lesson names it. What they are is the question.
5. What does the burn-rate guardrail do during an experiment?Define
A card naming the burn-rate alert, tagged to the Key Terms section, with what it does during an experiment left as a question mark.
The term as the lesson names it. What it does is the question.
6. What is a game day?Define
A card naming a game day, tagged to the Key Terms section, with what it is left as a question mark.
The term as the lesson names it. What it is is the question.
7. What is LitmusChaos?Define
A card naming LitmusChaos, tagged to the Key Terms section, with what it is left as a question mark.
The term as the lesson names it. What it is is the question.
8. What is Chaos Mesh?Define
A card naming Chaos Mesh, tagged to the Key Terms section, with what it is left as a question mark.
The term as the lesson names it. What it is is the question.
9. What sets Harness Chaos Engineering apart from the two CNCF tools?Define
A card naming Harness Chaos Engineering, tagged to the Key Terms section, with what sets it apart left as a question mark.
The term as the lesson names it. What sets it apart is the question.
10. What is a malformed prompt in chaos terms?Define
A card naming a malformed prompt, tagged to the Key Terms section, with what it is left as a question mark.
The term as the lesson names it. What it is is the question.
11. What is a KV eviction storm?Define
A card naming a KV eviction storm, tagged to the Key Terms section, with what it is left as a question mark.
The term as the lesson names it. What it is is the question.
12. Which cadence does the lesson recommend for running chaos?Define
A card naming the Cadence heading, tagged to The Concept section, with the schedule it recommends left as a question mark.
The heading as the lesson names it. The schedule is the question.
13. Which pairing of experiment and observation is correct for the five LLM-specific experiments?Explain
A card naming the five LLM-specific experiments: memory overload, network failure, provider outage simulation, malformed prompt, and KV eviction storm, with each one's own paired observation left as a question mark.
The five experiment names as the lesson lists them. No observation is paired to any of them.
14. Why is having SLIs and SLOs a prerequisite for chaos, rather than something you add later?Explain
A card stating that SLI/SLO is a required prerequisite set before any chaos run, not something treated as addable later, with why it must come first left as a question mark.
The setup exactly as given in the question. No reason is shown.
15. Why does the observability plane need trace-ID correlation on top of normal metrics?Explain
A card stating that the observability plane has normal metrics and dashboards, and adds trace-ID correlation on top, with why it adds the trace-ID tagging left as a question mark.
The setup exactly as given in the question. No reason is shown.
16. What is the job of the feedback loop in the four-plane design?Explain
A card stating that the four-plane design includes a feedback loop that runs after an experiment finishes, with what the loop's job is left as a question mark.
The setup exactly as given in the question. No destination is shown.
17. Why does the lesson begin with killing one decode replica under steady traffic?Explain
A card stating that the lesson's first experiment is killing one decode replica under steady traffic, with why it starts here left as a question mark.
The setup exactly as given in the question. No reason is shown.
18. Why does the lesson call for suppression windows around a chaos experiment?Explain
A card naming suppression windows as a guardrail active while the experiment runs, with why they exist left as a question mark.
The guardrail named exactly as the question gives it. No reason is shown.
19. During a chaos run in production, on-call is paged for errors and cannot tell which came from the experiment. Which safeguards were missing?Analyze
A card stating that on-call is paged for errors during a chaos run and cannot tell which errors came from the experiment, with which safeguards were missing left as a question mark.
The setup exactly as given in the question. No safeguard is named.
20. After an upstream provider returns 429s, your gateway retries, concurrency climbs, and the service runs out of memory. Which experiment would have caught this before users did?Analyze
A card stating that a provider returns 429s on calls, the gateway retries, concurrency climbs, and the service runs out of memory, with which experiment would catch this first left as a question mark.
The cascade exactly as the lesson describes it. No experiment is named.
21. A team has SLOs, dashboards, automated rollback, and runbooks, and plans chaos in production but has no on-call rota. What is the flaw?Analyze
A card stating that a team has SLOs, dashboards, automated rollback, and runbooks, lacks an on-call rota, and plans chaos in production, with what the flaw is left as a question mark.
The setup exactly as given in the question. No flaw is shown.
22. A team new to chaos wants a plan for its first month on a vLLM-based service. Which sequence fits the lesson?Design
A card stating that a team new to chaos engineering, running a vLLM-based service, is planning its first month, with which sequence fits left as a question mark.
The setup exactly as given in the question. No sequence is shown.
23. Your service runs entirely on virtual machines in a single public cloud, not on Kubernetes, and you want managed chaos with minimal setup. Which choice fits the lesson's tool list?Design
A card stating that a stack runs on virtual machines in a single public cloud with no Kubernetes present, and wants managed chaos with minimal setup, with which tool fits left as a question mark.
The setup exactly as given in the question. No tool is named.

Use It

code/main.py simulates three chaos experiments behind safety plane gates and reports which ones would trip the burn-rate abort.

Check: Use It0/5
1. In the toy runner, the pod kill experiment has a 0.002 induced error rate, a 0.0005 expected rate, and a 5% blast radius. What does it report?Apply
A card naming the pod-kill experiment with its own three real inputs: an induced error rate of 0.002, an expected baseline rate of 0.0005, and a blast radius of 5%, with the burn rate and status it reports left as a question mark.
The pod-kill experiment's own three real inputs from the toy runner. No burn rate or status is computed.
2. The provider 429 fallback experiment has a 0.015 induced error rate and a 30% blast radius against the same 0.0005 baseline. What does the runner do?Apply
A card naming the provider-429 fallback experiment with its own three real inputs: an induced error rate of 0.015, an expected baseline rate of 0.0005, and a blast radius of 30%, with the burn rate and status it reports left as a question mark.
The provider-429 experiment's own three real inputs from the toy runner. No burn rate or status is computed.
3. Your team runs the toy runner and sees a malformed-prompt experiment at 80.0x burn with a 10% blast radius. What happens, and what should you take from it?Apply
A card naming the malformed-prompt experiment with its own two given numbers, an 80.0x burn rate and a 10% blast radius, with what the run does and the lesson to draw from it left as a question mark.
The malformed-prompt experiment's own two given numbers. No status or lesson is drawn.
4. You change the provider 429 experiment in the toy runner so its blast radius is 15%, leaving the error rates as they are. What changes in the output?Apply
A card naming a modified provider-429 fallback experiment: its induced error rate of 0.015 and expected baseline rate of 0.0005 stay unchanged, while its blast radius changes from 30% to 15%, with the new burn rate and status left as a question mark.
The provider-429 experiment's own real rates, with only its blast radius changed. No new burn rate or status is computed.
5. You want to run a tokenizer-stall test that you expect to burn error budget fast. Which design keeps it safe and still teaches you something?Design
A card stating a tokenizer-stall test expected to burn error budget fast, with which design keeps it safe and still teaches you something left as a question mark.
The setup exactly as given in the question. No design is shown.

Ship It

This lesson produces outputs/skill-chaos-plan.md. Given your stack and maturity, it picks the first three experiments and the tooling.

Author's Note

I'd run main.py first and read the table before the prose. It runs three experiments (pod kill, provider 429 fallback, malformed prompt tokenizer stall) and only one of them aborts. The malformed prompt run has the highest burn rate in the table and still completes.

The reason is the gate. In the code it is a conjunction: burn rate above 2.0x expected and blast radius above 20%. The provider 429 experiment trips both, with a 30% blast radius. The tokenizer stall burns hot but touches 10% of traffic, so it runs to the end. The 2x rule from the guardrails list is only half of what the toy enforces, and I think the blast-radius half is the one worth copying. A hot experiment on a small slice is how you learn something cheaply.

I'd also keep the toy in proportion. The burn rate is a ratio of a hard-coded induced error rate to a hard-coded baseline, and the duration never enters the gate. Nothing here talks to a real service. It shows how the safety plane decides, not how to measure the inputs. For that you need the observability and SLO work from the prerequisites, and I'd do it before trying any of the five experiments against production.

Exercises

  1. Run code/main.py. Which experiment trips the burn-rate gate, and why?
  2. Design the first five chaos experiments for a vLLM-based RAG service. Include success criteria.
  3. Your burn-rate alert paused an experiment. How do you determine the root cause, chaos or natural?
  4. Argue whether chaos should run in production or only in staging. When is production the right answer?
  5. Name three LLM-specific failure modes that generic network chaos cannot reproduce.

Key Terms

TermWhat people sayWhat it actually means
SLI / SLO"service targets"Indicator + objective; required prerequisite
Blast radius"scope"Set of services / users affected by experiment
Burn-rate alert"budget gate"Fires when error-budget burn rate > 2x expected
Game day"monthly drill"Scheduled cross-team chaos exercise
LitmusChaos"CNCF workflow"Graduated CNCF Kubernetes chaos tool
Chaos Mesh"CNCF CRD"CNCF sandbox Kubernetes-native chaos
Harness CE"commercial AI-assisted"Harness chaos with AI recommendations
Malformed prompt"tokenizer bomb"Input that stalls tokenization
KV eviction storm"preemption cascade"Mass eviction triggering re-prefills

Further Reading

Code

Ships With