TAI BUI
Back to curriculum
LearnPython

Shadow Traffic, Canary Rollout, and Progressive Deployment for LLMs

LLM rollouts combine the hardest parts of software deployment: no unit tests, diffuse failure modes, and delayed signals. The sequence is (1) shadow mode: duplicate prod requests to the candidate model, log, and compare with zero user impact. It catches obvious distribution issues but is not a quality guarantee. (2) Canary rollout: shift traffic progressively, 1% → 10% → 25% → 50% → 75% → 100%, with gates at each step on latency percentiles, cost/request, error/refusal rate, output length distribution, and user-feedback rate. (3) A/B testing for distinct alternatives once stability is confirmed. Non-determinism does not go away: accuracy can vary by up to 15% across runs with identical inputs, from GPU FP non-associativity plus batch-size variance. Cost is a variable, so a 20% better model can be 3x more expensive per call. Rollback speed decides how bad a bad rollout gets, and if rollback needs a redeploy you are too slow. Policy lives in config/flags and the model lives in the registry with pinned digests, so rollback means flipping policy, reverting the threshold, and pinning the old model in seconds.

TIME: ~60 minutesPREREQUISITES: Phase 17 · 13 (Observability), Phase 17 · 21 (A/B Testing)

Learning Objectives

  • Distinguish shadow mode (zero-impact compare), canary (live traffic progressive), and A/B (stability-confirmed comparison).
  • Enumerate five LLM-specific canary metrics (latency, cost/request, error/refusal, output-length distribution, user feedback).
  • Explain why LLM non-determinism (up to 15%) changes what "stable" means in a rollout.
  • Design a rollback path that takes seconds (policy flip) not hours (redeploy).

The Problem

You ship a new model. Offline evals show a 3% accuracy gain, so you flip it on in production. Within 24 hours cost is up 40%, user thumbs-down is up 8%, and three customer tickets report "weird answers." You roll back, but the redeploy takes 3 hours and your weekend is gone.

Each of those failures had an earlier catch. Shadow mode would have shown the 40% cost spike before any user saw it. A canary would have stopped at 10% when thumbs-down moved. A policy-flag rollback would have taken 30 seconds. This lesson is about the discipline that fills the gap between "offline evals look good" and "real users are happy."

Warm-up0/3
1. You want to compare two LLM variants on live traffic. What must be in place before any comparison means anything?Warm-up
Two variants, A and B, each serving live requests, with an unlabeled dashed box asking what gets attached to each request. A line asks what must be in place before comparing them means anything.
Setup: the two variants taking traffic, nothing recorded yet. What needs to be attached per request is left open.
2. Why do monitoring dashboards usually show P99 latency next to the average?Warm-up
A scatter of 40 illustrative request latencies over one minute, nearly all clustered low and one far above the rest, with a dashed line marking the computed average sitting close to the low cluster. A line asks why dashboards show P99 latency next to that average.
Setup: the illustrative latencies and their average, nothing named as P99. Why the dashboard adds that second number is left open.
3. You measure an error rate on 20 requests and see 10%. Why should you not trust that number much?Warm-up
A grid of 20 request icons, 2 marked err and 18 marked ok, noted as a 10 percent error rate. A line asks why that number should not be trusted much.
Setup: the 20 requests and the 2 errors that make up the stated 10 percent, nothing about sample size. Why to distrust the number is left open.
Check: The Problem0/1
1. In the opening scenario the rollback took 3 hours. What was the underlying reason, and what removes it?Explain
The opening scenario in four steps: offline evals showing a 3 percent accuracy gain, shipped, 24 hours later cost up 40 percent with thumbs-down up 8 percent and 3 tickets, then a rollback that took 3 hours. A line asks what the underlying reason was and what removes it.
Setup: the scenario's own given timeline and numbers, with no cause named. The underlying reason and its fix are left open.

The Concept

Shadow mode

Candidate receives the same requests as production; outputs are logged, not returned to users. Zero user impact. Log:

  • Output content (diff against production).
  • Token counts (cost delta).
  • Latency.
  • Refusal and error.

Shadow catches cost blow-ups, length regressions, obvious refusal changes, and hard errors. It does not catch the quality difference users would perceive. Treat it as a smoke test, not a quality test.

Canary rollout

Shift traffic progressively and gate each step. A typical progression: 1% → 10% → 25% → 50% → 75% → 100%. Gate on 5 metrics at each step:

  1. Latency percentiles: P50, P95, P99. Breach: canary has P99 > 1.5x baseline.
  2. Cost per request: blended $. Breach: >20% above baseline.
  3. Error / refusal rate: 5xx plus explicit refusals. Breach: 2x baseline.
  4. Output length distribution: mean + P99. Breach: distributional shift.
  5. User-feedback rate: thumbs-down / ticket filings. Breach: 1.5x baseline.

Non-determinism is the new variance

Identical inputs can produce different outputs. Three causes:

  • GPU FP non-associativity (floating-point reduction order varies by batch).
  • Batch-size variance (same prompt in a batch of 128 vs batch of 16).
  • Sampling (temperature > 0).

Measured: up to 15% accuracy variation run-to-run on identical eval sets. "Stable" in a rollout means metrics sit within expected variance of baseline, not that they match it. Set gates above the noise floor.

Cost is a variable

A 20% better model can be 3x more expensive per call, which is why cost/request is one of the five gates. A "better" model that breaks unit economics is a rollback case.

Rollback is the weapon

  • Policy flag (feature flag system): flip the percentage in config. Takes seconds.
  • Model pinning (registry digest): a pinned model does not auto-upgrade.
  • Rollback means reverting the flag and setting the pinned digest back to the previous one. Seconds, not hours.

If your stack needs a redeploy to roll back, fix that before you start rolling out.

Tooling

Argo Rollouts / Flagger: Kubernetes progressive delivery controllers. They integrate with Istio/Linkerd weighted routing.

Istio weighted routing: traffic split at the service-mesh level.

KServe / Seldon Core: model serving with built-in canary.

Feature flags: LaunchDarkly, Flagsmith, Unleash. A policy-level flip with no redeploy.

Metrics cadence

Canary gates check every 5-15 minutes depending on traffic volume. 1% traffic with 10 req/min gives 50-150 data points per window, which is enough for latency but noisy for user feedback. 10% gives ~10x more. Pause long enough at each step to accumulate the samples the gate needs.

The A/B step is optional

If the new model is distinctly different (behavior, cost curve, tone), A/B test it at 50% after the canary passes. If it is just an improved version, go to 100% once the canary gates pass.

Numbers you should remember

  • Canary progression: 1% → 10% → 25% → 50% → 75% → 100%.
  • Non-determinism ceiling: up to 15% run-to-run variance on identical inputs.
  • Five canary metrics: latency, cost, error/refusal, output length, user feedback.
  • Cost gate: >20% above baseline is a breach.
  • Rollback: seconds, not hours.
Check: The Concept0/24
1. What is shadow mode in a model rollout?Define
One request duplicated to two destinations: the solid path goes request, production, user; the dashed path goes request, candidate, log, with no arrow from the candidate or the log back to the user. A line asks what this setup is called and what it is.
Setup: where each copy of the request goes, drawn as arrows only. What the setup is called is left open.
2. What canary traffic progression is typical?Define
A rising line from a small traffic share to all users, with one gate-check marker at an unlabeled point partway along it. A line asks what progression of shares is typical and what happens along it.
Setup: traffic rises from a small share to everyone, with gate checks happening somewhere along the way. The actual stages and their values are left open.
3. Which set of metrics gates each canary step?Define
A card naming Gates as something checked before a canary advances to its next traffic share. A line asks which set of metrics gates each step.
Setup: that gates exist and block progression. The actual set of metrics is left open.
4. What does non-determinism mean for an LLM in production?Define
A card naming non-determinism and noting where it falls in the lesson's own section order, after the canary gates and before cost and rollback. A line asks what it means for an LLM in production.
Setup: the term and its place in the lesson, nothing about what it means. What it means is left open.
5. What is a policy flag in a rollout?Define
A card naming policy flag and placing it in the Rollback section alongside Model pin. A line asks what it is.
Setup: the term, among the company it keeps. What it is is left open.
6. What is a model pin?Define
A card naming model pin and placing it in the Rollback section alongside Policy flag. A line asks what it is.
Setup: the term, among the company it keeps. What it is is left open.
7. What is Argo Rollouts?Define
A card naming Argo Rollouts among the Tooling section's other named tools, Flagger, KServe or Seldon Core, Istio weighted routing, and feature flags. A line asks what it is.
Setup: the name, among its sibling tools. What it is is left open.
8. What does KServe provide?Define
A card naming KServe among the Tooling section's other named tools, Seldon Core, Argo Rollouts or Flagger, Istio weighted routing, and feature flags. A line asks what it provides.
Setup: the name, among its sibling tools. What it provides is left open.
9. What does Istio weighted routing do in a canary?Define
A card naming Istio weighted routing among the Tooling section's other named tools, Argo Rollouts or Flagger, KServe or Seldon Core, and feature flags. A line asks what it does in a canary.
Setup: the name, among its sibling tools. What it does is left open.
10. Which breach thresholds are set for the latency, error/refusal, and user-feedback gates?Define
A card naming three of the five gates, latency, error or refusal rate, and user feedback, each described as breaching at a multiple of its own baseline, with no multiples given. A line asks what multiple sets off each.
Setup: the three gate names, no numbers. The breach multiples are left open.
11. What is shadow mode good for?Explain
A card naming shadow mode and noting it runs before the canary ramp in the lesson's own rollout order. A line asks what it is good for.
Setup: where it sits in the sequence, nothing about its scope. What it is good for is left open.
12. A teammate says shadow mode is enough and the canary can be skipped. What does that miss?Explain
Two paths side by side, shadow output going only to a log and canary output going to real users, with no other facts shown. A line says a teammate wants to skip the canary and asks what relying on shadow alone misses.
Setup: only where each path's output goes. What skipping the canary misses is left open.
13. How much run-to-run accuracy variation can identical inputs show on an LLM eval, and what does it mean for gates?Explain
A card naming non-determinism and how it is measured, by repeating identical inputs on the same model. A line asks how much run-to-run accuracy variation can show up and what it means for gates.
Setup: how the variation is measured, no figure given. The amount and what it means for gates are both left open.
14. Which three causes drive LLM non-determinism?Explain
A card saying the lesson names exactly three causes of non-determinism, with none of the three listed. A line asks which three.
Setup: that there are three named causes. Which three they are is left open.
15. Why is cost a gate next to latency and quality?Explain
A card naming cost per request as one of the five canary gates, alongside the other four left unnamed. A line asks why it is a gate next to latency and quality.
Setup: that cost is one of the five gates, no reasoning given. Why it is a gate is left open.
16. When is an A/B step at 50% worth adding after the canary passes?Explain
The current model and a candidate that already passed the canary, each with tone and cost curve left as blanks. A line says the team adds an A/B step at 50% past the canary and asks when that is worth doing.
Setup: two models with their traits left blank. When the extra step is worth adding is left open.
17. What does a correct rollback design look like?Explain
A card naming rollback as something a team reaches for right after a gate trips during a canary. A line asks what a correct design looks like.
Setup: when rollback gets used, nothing about the mechanism. What a correct design looks like is left open.
18. Which statement about gate cadence is accurate?Explain
A clock icon labeled gates check periodically, next to a canary stage box noted as holding a small traffic share right now. A line asks which statement about that checking cadence is accurate.
Setup: that checks happen periodically at a small-traffic stage, with no interval or sample counts shown. Which statement about cadence is accurate is left open.
19. The thumbs-down gate halts a 1% stage, but a later window with the same model passes. At about 10 requests per minute, which diagnosis fits?Analyze
Two windows at the same 1 percent stage with the same model and about 10 requests per minute: window 1 halts on the thumbs-down gate, window 2 passes. A line asks which diagnosis fits.
Setup: the two windows and their outcomes, with no explanation given. Which diagnosis fits is left open.
20. A team sets every gate at 1.05x baseline. A healthy candidate keeps halting at random stages. Which assumption is wrong?Analyze
Every gate set to 1.05 times baseline, next to a note that a healthy candidate keeps halting at random stages. A line asks which assumption is wrong.
Setup: the gate setting and the observed pattern, with no diagnosis given. Which assumption is wrong is left open.
21. Shadow mode showed cost, latency, refusals and errors all in range, so a team launched straight to 100%. Users then reported worse answers. What was the wrong assumption?Analyze
A table of shadow mode's four readings before launch, cost, latency, refusals and errors, all in range. A line says the team launched straight to 100% and users then reported worse answers.
Setup: the four given shadow readings and what happened after launch. What was wrong is left open.
22. A candidate's mean output length matches baseline, but its P99 length has doubled because some replies ramble. Which gate design misses this?Analyze
Two side-by-side bar pairs: mean output length unchanged between baseline and candidate, and P99 output length doubled from main.py's own baseline of 450 to 900. A line asks which gate design misses this.
Setup: the given mean (unchanged) and P99 (doubled from the real 450 baseline to 900) readings, with no gate design named. Which design misses the shift is left open.
23. Your new model has a distinctly different tone and a different cost curve from the current one. Which rollout plan fits?Design
A card describing a new model with a distinctly different tone and a distinctly different cost curve from the current one. A line asks which rollout plan fits.
Setup: the two given differences, no plan shown. Which plan fits is left open.
24. Your stack rolls back by rebuilding and redeploying, which takes hours. You are about to start a rollout. What comes first?Design
A card describing a stack where rollback means rebuilding and redeploying, which takes hours. A line says the team is about to start a canary rollout and asks what comes first.
Setup: the given slow rollback path and the plan to start rolling out. What comes first is left open.

Use It

code/main.py simulates a canary rollout with injected regressions. It reports which stage the rollout halts at and which gate triggered.

Two simplifications to know about. The latency gate checks P99 only (no P50 or P95), and the output-length gate is P99 above 1.4x baseline rather than a full distribution comparison. Run it, then compare the six scenarios against the gate table in The Concept.

Check: Use It0/10
1. Your canary is at 25% traffic. Cost per request is 1.3x baseline, and latency, errors, length and thumbs-down are all flat. What should happen?Apply
A canary at a 25 percent traffic share, with cost per request given as 1.3 times baseline and latency, errors, length and thumbs-down all noted as flat. A line asks what should happen.
Setup: the stage and the given metric readings, no gate rule stated. What should happen is left open.
2. In main.py the baseline P99 latency is 900 ms and the latency gate is 1.5x. Which measured P99 halts a stage?Apply
main.py's own constants: a baseline P99 latency of 900 milliseconds and a latency gate multiplier of 1.5, not yet combined. A line asks which measured P99 halts a stage.
Setup: the two real constants, not yet multiplied together. Which reading halts a stage is left open.
3. In main.py, cost_mult is 1.10 and the noise multiplier is drawn between 0.92 and 1.08. Can the 1.2x cost gate ever trip?Apply
main.py's own values for this scenario: a cost multiplier of 1.10, noise drawn between 0.92 and 1.08, and the cost gate at 1.2, not yet combined. A line asks whether the 1.2x cost gate can ever trip.
Setup: the three given values, not yet combined. Whether the gate can trip is left open.
4. In main.py, latency_mult is 1.8 and noise is drawn between 0.92 and 1.08. What happens?Apply
main.py's own values for this scenario: a latency multiplier of 1.8, noise drawn between 0.92 and 1.08, a baseline P99 of 900 milliseconds, and a latency gate of 1.5, not yet combined. A line asks what happens.
Setup: the four given values, not yet combined into a reading or a limit. What happens is left open.
5. main.py runs cost_mult=1.15 with thumbs_down_mult=1.45. What does its printed output show?Apply
main.py's own given values for this scenario, a cost multiplier of 1.15 and a thumbs-down multiplier of 1.45, with no printed output shown yet. A line asks what the printed output shows.
Setup: the two given multipliers, nothing run yet. What the printed output shows is left open.
6. In main.ts, after engine.promote("candidate-digest-v2", 0.1), roughly how many of 1,000 pick() calls choose the candidate?Apply
main.ts's own PolicyEngine source: promote sets the pinned digest and rolloutPct, and pick compares a random draw against rolloutPct. A line says promote is called with 0.1 and asks how many of 1000 pick calls choose the candidate.
Setup: the real promote and pick source, and the given 0.1 argument. How many picks choose the candidate is left open.
7. In main.ts, what does engine.rollback() with no argument do?Apply
main.ts's own rollback source: when no baselineDigest argument is given, it skips reassigning baselineDigest, then sets pinnedDigest to it and rolloutPct to 0. A line asks what engine.rollback() with no argument does.
Setup: the real rollback source, with no argument supplied. What it does is left open.
8. In main.ts every shadow sample has candidateCost 0.028 against baselineCost 0.02, with equal latencies. What does shadowEvaluate return?Apply
A table of the given per-sample values, candidateCost 0.028, baselineCost 0.02, and equal latencies, for every shadow sample, not yet combined into a delta. A line asks what shadowEvaluate returns.
Setup: the given per-sample values, not yet averaged. What shadowEvaluate returns is left open.
9. In main.ts, canaryRollout returns { promoted: false, stagesAdvanced: 2 }. What happened?Apply
main.ts's own canaryRollout loop source, returning stagesAdvanced as the loop index i at the moment a breach is found. A line says it returns stagesAdvanced 2 and asks what happened.
Setup: the real loop source and the given return value. What that value means is left open.
10. Design a shadow alert that catches a 40% cost spike but ignores per-request price jitter. Which rule fits?Design
A scatter of per-request cost deltas with illustrative jitter scattered around a sustained plus 40 percent line. A line says the alert needs to catch this spike but ignore ordinary per-request jitter and asks which rule fits.
Setup: the jittery per-request readings and the sustained 40 percent shift they average to, no alert rule stated. Which rule fits is left open.

Ship It

This lesson produces outputs/skill-rollout-runbook.md. Given a candidate model, a baseline, and a risk tolerance, it designs a shadow→canary→100% plan.

Author's Note

I'd run main.py before trusting any of the multipliers above. It walks six stages (1%, 10%, 25%, 50%, 75%, 100%) against five gates and prints PASS or HALT per stage, plus a rollback line when a gate trips. The scenarios are a clean promotion, a 10% cost regression, a 25% cost regression, an 80% latency regression, a 60% thumbs-down regression, and a mix of 15% cost creep with a 45% thumbs-down rise.

The 10% cost case is the one worth staring at. It passes at every stage because the simulator adds up to 8% noise to each metric, and 1.10 times 1.08 is still under the 1.2 cost gate. The 15% cost regression sits closer to the line, and in the last scenario it halts at 25% with cost and thumbs-down both tripping. (main.ts runs the same scenarios on a different random generator, so it halts that last one at the 1% stage on thumbs-down alone.) The gates are not strict thresholds on the true value. They are thresholds on a noisy reading, and that is the same reason the gates need to sit above the noise floor.

I'd also keep the toy in proportion. Its noise width is the same at every stage, so 1% traffic reads as cleanly as 75%. Real canaries get noisier as traffic drops, and user feedback suffers most, as the metrics cadence section describes. The simulator also has no shadow phase and no A/B step, and its rollback is a printed line rather than a flag flip. Exercise 3 is where you build the part it leaves out.

Exercises

  1. Run code/main.py. Inject a 25% cost regression. At which stage does the canary halt, and why that stage? Then try 15% and compare.
  2. Your new model has 3% accuracy gain offline but cost/request is +18%. Is it a ship? Depends on the policy, so write both paths.
  3. Design a rollback that takes under 60 seconds end-to-end. List the required infrastructure.
  4. Non-determinism shows ±7% on your eval. Set canary gates so you don't false-alarm. What multipliers do you use?
  5. Shadow mode catches a 40% cost spike before canary. Write the alert rule that fires in shadow.

Key Terms

TermWhat people sayWhat it actually means
Shadow mode"duplicate to new"Zero-impact send-to-candidate for logging
Canary"progressive traffic"Gradual user-exposed rollout with gates
Gates"rollout checks"Metric thresholds that block progression
Non-determinism"LLM variance"Irreducible run-to-run differences
Policy flag"flag flip rollback"Config-level rollback, seconds not hours
Model pin"registry digest"Immutable reference to a model version
Argo Rollouts"K8s progressive"Kubernetes-native canary/rollback controller
KServe"inference K8s"Model serving with canary primitives
Istio weighted"mesh split"Service-mesh traffic splitter

Further Reading

Code

Ships With