LearnPython

Model Routing as a Cost-Reduction Primitive

A dynamic broker evaluates every request (task type, token length, embedding similarity, confidence) and sends simple queries to a cheap model, escalating complex ones to a frontier model. This is also called model cascading. Production case studies show 20-60% cost reduction at iso-quality across US/UK/EU deployments, and a 30% routing efficiency improvement on high-volume SaaS turns into six-figure annual savings. The 2026 context is that LLM inference prices dropped ~10x per year: a GPT-4-class token went from $20/M to ~$0.40/M from late 2022 to 2026. Most of the drop is better serving stacks (Phase 17 · 04-09), not hardware. Routing is how you convert that price drop into margin without product regression. The failure mode is cheap-model drift. The route pushes 40% to a weaker model, quality drops 3-5% on reasoning tasks, and no one notices for a quarter. Gate routes by online quality metrics, not just offline eval sets.

TIME: ~60 minutesPREREQUISITES: Phase 17 · 01 (Managed LLM Platforms), Phase 17 · 19 (AI Gateways)

Learning Objectives

Explain model cascading: cheap-first with confidence check, escalate on low confidence.
Enumerate the four routing signals (task classification, prompt length, embedding similarity to known-hard set, self-confidence from first-pass).
Compute expected blended cost at target routing split and quality loss tolerance.
Name the drift-monitoring metric (online quality gate) that catches cheap-model creep.

The Problem

Your service costs $80k/month on GPT-5. Your analytics show 70% of queries are simple: "what time is it in Paris?" "rephrase this sentence." A Haiku-class model handles those well at 3% of the cost. The other 30% need GPT-5's reasoning: coding, math, multi-step planning.

If you route the 70% to cheap and 30% to expensive, your bill drops ~65% at the same product quality. This is routing. The hard part is building the broker without regressing quality.

Warm-up0/2

1. What does an AI gateway do in a production LLM stack?Warm-up

An app connected through an unlabeled middle layer to three separate hosted model providers. A line asks what that middle layer actually does. — Setup: the app, an unlabeled layer, and three providers. What the layer does is left open.

2. What do managed LLM platforms sell you?Warm-up

An invoice for GPT-5 API usage totaling $80,000 for the month, the same figure the lesson's Problem section opens with. A line asks what that payment is actually for. — Setup: the lesson's own $80,000 monthly bill. What it pays for is left open.

Check: The Problem0/3

1. What is model routing?Define

Two available models, a cheap one and a frontier one, with a note that for each separate incoming request something decides which one handles it. A line asks the name for that. — Setup: the two available models. The name for the per-request decision is left open.

2. 70% of traffic goes to a model at 3% of the frontier price and 30% stays on frontier. Roughly what share of the original bill remains?Apply

Setup: the two given rows of the split, not yet combined. The overall remaining share is left open.

3. Your users are 30% enterprise with complex queries and 70% free tier with simple ones. Which design fits best?Design

A bar chart of the product's user base, 30 percent enterprise with complex queries and 70 percent free tier with simple ones. A line asks which routing design fits this split best. — Setup: the given 30/70 user split. Which design fits it is left open.

The Concept

Four routing signals

Task classification: simple/complex/codegen/math/chat. It can be a rules-based classifier, a small LLM (Haiku-class at $0.25/M), or embedding similarity to labeled buckets. The output is a route: cheap, balanced, or frontier.
Prompt length: prompts over 4K tokens often need frontier for coherence. Prompts under 500 tokens usually don't.
Embedding similarity to known-hard set: if the query is close (cosine > 0.88) to a known-hard bucket, escalate to frontier directly.
Self-confidence from first-pass: send to cheap; if model's log-probs show low confidence OR it refuses OR outputs hedging language, retry on frontier. This adds P95 latency on ~10% of traffic but saves 50%+ on the other 90%.

Three patterns

Pre-route (classifier up front) adds ~5-10ms of latency and is the fastest overall.

Cascade (cheap-first, escalate on low confidence) runs at ~1.2x median latency (cheap run plus verify) and ~2x on escalated requests. It has the best quality floor.

Ensemble route runs cheap and frontier in parallel for a sample and lets a reward model pick. It has the highest quality and the highest cost, so save it for critical A/B tests.

Implementation

AI gateways (Phase 17 · 19) expose routing. LiteLLM has a router config with fallback and cost-routing, Portkey has guards plus routing, and Kong AI Gateway has plugin-based routing. OpenRouter's model marketplace exposes a recommendation API.

Open-source: RouteLLM (LMSYS), Not Diamond (commercial), Prompt Mule.

The 2026 price curve

Model class	Late 2022	2026	Change
GPT-4-level quality	~$20/M	~$0.40/M	50x cheaper
Frontier (GPT-5, Claude 4)	n/a	~$3-10/M	new tier

Most of the improvement is serving efficiency. The core lessons in Phase 17 · 04-09 turned into provider-side cost drops, and routing lets you capture those gains at the app layer instead of waiting for all your users to migrate to the cheap tier.

Drift is the real risk

Say your route sends 40% to the cheap model. Over six months the task distribution shifts: users get more sophisticated and ask longer questions. The router doesn't notice because its classifier was trained on Q1 data, so quality drops silently and nobody complains loudly enough. You find out when you lose a competitor benchmark.

Gate routes by online quality metrics:

User thumbs-up / thumbs-down per route.
Automated LLM-judge on a held-out sample (5%) per route.
Escalation rate: if the cascade is kicking up-route more than 30% of the time, the cheap model is being over-routed.
Refusal rate per route.

Numbers you should remember

2026 routing savings at iso-quality: 20-60% case studies.
LLM price drop 2022-2026: ~10x per year aggregate.
GPT-4-level 2022 vs 2026: ~$20/M → ~$0.40/M.
Cascade latency impact: ~1.2x median, ~2x escalated (~10% of traffic).

Check: The Concept0/23

1. What is the core idea of model cascading?Define

A card naming model cascading as one of three patterns the lesson covers, alongside pre-route and ensemble route. A line asks what specifically happens in a cascade. — Setup: the term, named alongside its two sibling patterns. What it actually does is left open.

2. Which pattern adds roughly 5-10ms up front and is fastest overall?Define

A single timeline showing a fixed 5 to 10 millisecond cost added once, up front, before a request proceeds. A line asks which of the three named patterns this is, the fastest overall. — Setup: the given timing fact alone. Which pattern it names is left open.

3. What does the ensemble route pattern do?Define

A card naming ensemble route and noting the lesson reserves it for critical A/B tests, unlike the other two patterns. A line asks what it actually does. — Setup: the term and where the lesson says to use it. What it actually does is left open.

4. In a cascade, what does escalation rate measure?Define

A card naming escalation rate as a number people watch while a cascade runs. A line asks what it measures. — Setup: the term alone, situated in a running cascade. What it measures is left open.

5. What is RouteLLM?Define

A card naming RouteLLM and listing it among LiteLLM, Portkey, Kong AI Gateway, OpenRouter, Not Diamond and Prompt Mule. A line asks what kind of thing it is. — Setup: the name, in the company of the other named tools. What kind of thing it is is left open.

6. What is Not Diamond?Define

A card naming Not Diamond and listing it among RouteLLM, LiteLLM, Portkey, Kong AI Gateway, OpenRouter and Prompt Mule. A line asks what kind of thing it is. — Setup: the name, in the company of the other named tools. What kind of thing it is is left open.

7. What is cheap-model drift in routing?Define

A card naming cheap-model drift as a failure mode the lesson warns about. A line asks what it is. — Setup: the term alone. What it is is left open.

8. What is an online quality gate in routing?Define

A card naming an online quality gate as a guard the lesson recommends against drift. A line asks what it does. — Setup: the term alone. What it does is left open.

9. Which set of signals does the Concept section list for routing decisions?Explain

One request with several unlabeled dotted lines fanning in from different directions, representing more than one kind of information feeding the routing decision. A line asks which kinds, specifically. — Setup: an unlabeled, uncounted fan of inputs into one request. Which specific signals they are is left open.

10. Why does a cascade check the cheap model's first-pass answer before returning it?Explain

A cheap model's answer flowing into an unlabeled check before it is returned to the caller. A line asks what is checked and why, before the answer goes out. — Setup: the check happens, placed before the answer is returned. What it checks and why is left open.

11. Why does a close match to a known-hard bucket send a query straight to frontier?Explain

Setup: the match and the skip, both given. The reason for skipping is left open.

12. What is the expected latency profile of a cascade router?Explain

One request moving through a cheap-model run and a confidence check, with a frontier run shown only as a dashed, conditional extra step if escalated. A line asks the typical latency multiples, median and escalated, compared to one plain model call. — Setup: the conditional extra step, unresolved. Both latency multiples are left open.

13. Why is pre-route the fastest of the three routing patterns?Explain

A card naming pre-route and stating it is already known to be the fastest of the three patterns. A line asks why. — Setup: the given fact that pre-route is fastest. The reason is left open.

14. Why does the lesson reserve ensemble routing for critical A/B tests?Explain

A cost versus quality axis with ensemble route marked in the high-cost, high-quality corner. A line asks why the lesson reserves it for critical A/B tests only. — Setup: where ensemble route sits on cost and quality. The reason it stays limited to critical A/B tests is left open.

15. Provider-side serving efficiency keeps cutting model prices. What does routing let your app do about that?Explain

A line falling from about $20 per million tokens in late 2022 to about $0.40 per million tokens in 2026, the lesson's own price curve. A line asks what the app can do about most of that drop, which comes from the provider's own serving efficiency, right now. — Setup: the lesson's real price curve. What the app can do about it right now is left open.

16. Which guard catches routing drift?Explain

A cheap model's route running in production, next to an unlabeled box watching it. A line asks what keeps checking whether the route is still good. — Setup: a live route and an unlabeled watcher beside it. What actually does the watching is left open.

17. Why can a router's own classifier not notice drift?Explain

A classifier trained once, with time passing before new traffic arrives, and a note that the classifier itself keeps acting the same way toward it. A line asks why it doesn't notice anything changed. — Setup: the classifier's one-time training and the new traffic that later arrives. Why it fails to notice anything changed is left open.

18. A cascade escalated 700 of 2,000 requests this week. What does the lesson's guideline conclude?Apply

A bar chart of this week's 700 escalated requests against 1,300 that did not, out of 2,000 total. A line gives the lesson's over-routing rule with its percent threshold blanked, and asks where this week lands against it. — Setup: the raw counts for this week, and the rule with its threshold left blank. Where this week lands against the rule is left open.

19. A new query has cosine similarity 0.91 to a known-hard bucket. What does the similarity signal do with it?Apply

A 0-to-1 similarity axis with a new query marked at 0.91 and the known-hard region's own starting point left unlabeled. A line asks where that region starts and what the signal does with this query. — Setup: the query's own similarity value, 0.91, given. The threshold and the resulting action are both left open.

20. One prompt is 6K tokens and another is 300 tokens. What does the prompt-length signal suggest?Apply

Setup: the two given prompt lengths. The signal's thresholds and each prompt's verdict are left open.

21. A cascade has run for six months. Escalation rate climbed from 8% to 22% and thumbs-down on the cheap route is rising. What do you do first?Analyze

A line climbing from 8 percent to 22 percent escalation rate over six months, with a note that thumbs-down on the cheap route is rising over the same stretch. A line asks what to do first. — Setup: the real climb in escalation rate and the given thumbs-down trend. What to do first is left open.

22. A chat product has a tight latency budget and cannot afford a retry on any request. Which routing pattern fits?Design

A card listing two constraints for a chat product, a tight latency budget and no room for a retry on any request. A line asks which routing pattern fits. — Setup: the two given constraints. Which pattern fits is left open.

23. Wrong answers are costly in your product, but most requests are easy. Which routing pattern fits?Design

A card listing two facts about a product, that wrong answers are costly and that most requests are easy. A line asks which routing pattern fits. — Setup: the two given facts. Which pattern fits is left open.

Use It

code/main.py simulates an all-frontier baseline, pre-route, and cascade on a mixed workload of 1000 requests. It reports blended cost, savings against the baseline, mean quality, and escalation count.

Check: Use It0/5

1. A cascade handles 1,000 requests. Frontier costs 1 unit per request, cheap costs 0.03, and 10% of requests escalate. What is the total cost?Apply

A table of the stem's own given numbers for a cascade: 1,000 requests, a cheap cost of 0.03 paid on every request, a frontier cost of 1 paid only on the 100 escalated requests. A line asks for the total cost. — Setup: all four given numbers, not yet combined. The total cost is left open.

2. In the toy workload there are 100 simple, 100 medium and 100 hard requests. Hard ones always escalate in the cascade and half of the medium ones do. How many escalate?Apply

A bar chart for a 100 simple, 100 medium, 100 hard workload under the stem's own given rule: 0 simple, 50 medium (half) and 100 hard (all) escalate, each a direct reading of the stated rule. A line asks the total that escalate. — Setup: the rule and each group's own escalation count, worked from it. Only the combined total is left open.

3. With the same 100 simple, 100 medium and 100 hard requests, how many does the toy pre-route send to frontier?Apply

A bar chart for the same 100 simple, 100 medium, 100 hard workload under pre-route's own rule: 0 simple, and 100 each of medium and hard go to frontier, each a direct reading of the stated rule. A line asks the total sent to frontier. — Setup: the rule and each group's own frontier count, worked from it. Only the combined total is left open.

4. A cascade's monthly bill comes out higher than the all-frontier baseline. Which finding best explains it?Analyze

The general cost shape of a cascade, cheap cost on every request plus frontier cost on escalated ones, with a note that this month the cascade bill came out above the all-frontier baseline. A line asks which finding best explains that. — Setup: the general cost shape and the given fact that the bill exceeded baseline this month. Which finding explains it is left open.

5. The simulator shows cascade saving more than pre-route, and a teammate wants to ship that split unchanged. What is the main flaw?Analyze

The lesson's own real terminal output: NO_ROUTE, PRE_ROUTE and CASCADE rows with their real cost, savings and escalation counts, where CASCADE saves more than PRE_ROUTE. A line says a teammate wants to ship that split unchanged and asks the main flaw. — Setup: the real simulator output. The flaw in trusting it unchanged is left open.

Ship It

This lesson produces outputs/skill-router-plan.md. Given workload and quality budget, picks a routing pattern and signals.

Author's Note

I'd run main.py first and read the three result rows before the prose. It is stdlib only, builds 1000 requests from a fixed seed, and prints cost, savings against the all-frontier baseline, mean quality, and escalation count for no routing, pre-route, and cascade.

The row worth staring at is cascade. It pays for the cheap model on every request and adds the frontier cost on top for the ones that escalate, so the escalation count is what decides whether it saves money. In the toy, hard requests always escalate and half of the medium ones do. Pre-route sends every non-simple request straight to frontier, so it saves less here. Cascade gives up a little quality for that extra saving, and the extra latency on escalated requests is the price you pay for the quality floor.

Keep the toy in proportion. The pre-route "classifier" reads the difficulty label straight off the request, so it is right every time, which flatters it. The cheap-model quality numbers are a hardcoded table per difficulty, and the cascade's confidence check is a coin flip on medium requests. Nothing in it measures drift. That part you have to build yourself, with the online quality gate from the Concept section, before you trust any split you get from a simulator.

Exercises

Run code/main.py. At what accuracy floor does cascade beat pre-route?
Your user base is 30% enterprise (complex queries), 70% free tier (simple). Design the routing split. What online metric gates it?
A route drops quality by 2% but saves 40%. Is that a ship? It depends on the product, so argue both sides.
Implement a confidence check using logprobs from OpenAI / Anthropic APIs. What's the threshold you start with?
Over six months, escalation rate climbs from 8% to 22%. Diagnose three causes and the fix for each.

Key Terms

Term	What people say	What it actually means
Model routing	"cost broker"	Dynamic choice of model per request
Model cascade	"cheap-first escalate"	Run cheap, fall through to frontier on low confidence
Pre-route	"classify first"	Classifier up front; no re-run
Ensemble route	"parallel pick"	Run multiple, reward-model picks best
Escalation rate	"uprouted %"	Fraction of cascade requests that escalated
RouteLLM	"LMSYS router"	OSS router library
Not Diamond	"commercial router"	SaaS model-routing product
Drift	"cheap creep"	Distribution shift without router noticing
Online quality gate	"live check"	Automated LLM-judge sampling live traffic