TAI BUI
Back to curriculum
LearnPython

LLM Routing Layer — LiteLLM, OpenRouter, Portkey

Provider lock-in is expensive, and different tool-calling workloads suit different models. A routing gateway gives you one API surface plus retries, failover, cost tracking, and guardrails. Three archetypes dominate 2026: LiteLLM (open-source, self-hosted), OpenRouter (managed SaaS), and Portkey (production-grade, open-sourced in March 2026). This lesson names the decision criteria and walks through a stdlib routing gateway.

TIME: ~45 minutesPREREQUISITES: Phase 13 · 02 (function calling), Phase 13 · 17 (gateways)

Learning Objectives

  • Distinguish self-hosted, managed, and production-grade routing options.
  • Implement a fallback chain that retries on provider failures in a defined priority order.
  • Track per-request cost and token usage across providers.
  • Decide between LiteLLM, OpenRouter, and Portkey for a given production constraint.

The Problem

Five scenarios where provider routing matters:

  1. Cost. Claude Sonnet costs 3x what Haiku costs. Haiku is enough for a triage task, and Sonnet is worth it for synthesis. Route per request.

  2. Failover. OpenAI has a bad hour and requests fail. You want automatic fallback to Anthropic without a redeploy.

  3. Latency. A live chat UI needs a fast time-to-first-token. A batch summarizer does not. Route by latency SLA.

  4. Compliance. EU users must stay in EU regions, so you route by region.

  5. Experimentation. You A/B two models on the same workload and route by test bucket.

Hand-coding this per integration gets repetitive fast. A routing gateway gives you one OpenAI-compatible API and handles the rest.

Warm-up0/4
1. In function calling, which side actually executes the tool after the model emits a tool call?Warm-up
A model box with an arrow to a card holding a structured tool call named get_weather with the argument city set to Oslo, and a dashed question box asking who runs get_weather.
Typed example with illustrative names, not captured output.
2. In the MCP gateways lesson, what does a gateway do for the clients that connect to it?Warm-up
Three client boxes with arrows into one gateway box, and a dashed question box labelled clients get.
Typed schematic, not captured output.
3. A client calls an HTTP API and gets back a 503 status code. What does that status class usually indicate?Warm-up
A client box and an HTTP API box joined by a request arrow and a response arrow, a card reading HTTP status 503 between them, and a dashed question box about what the 5xx class indicates.
4. A service keeps the same endpoint and request format but swaps the software running behind it. What happens to the clients?Warm-up
A clients box with an arrow to a box marked same endpoint and same request format, then to an old software box drawn dashed and a new software box, and a dashed question box asking what clients must now do.
Typed schematic, not captured output.
Check: The Problem0/4
1. What is a routing gateway?Define
A box for your app sending a request to a box named routing gateway and receiving a reply, with a dashed question box under the gateway.
Typed schematic, not captured output.
2. Why does the lesson prefer a gateway to hand-coding provider logic in each integration?Explain
A code card of one hand-written integration with a try and except around two provider calls, a cost line and a region check, and a dashed question box asking why the lesson prefers a gateway.
Code is a typed illustration, not from main.py.
3. EU users must stay in EU regions, and you also need failover when a provider has a bad hour. Which design fits?Design
A dashed boundary labelled EU regions containing an EU users box, two requirement lines, stay in EU regions and failover on a bad hour, and a dashed question box asking for the design.
Typed schematic, not captured output.
4. A live chat UI needs fast first tokens, while a nightly batch summarizer does not care about latency. Which gateway setup fits both?Design
A live chat UI box that needs fast first tokens and a nightly batch summarizer box that does not care about latency, both feeding one gateway box, and a dashed question box asking for the setup that fits both.
Typed schematic, not captured output.

The Concept

OpenAI-compatible proxy shape

Most clients already speak the OpenAI shape. The routing gateway exposes /v1/chat/completions, accepts the OpenAI schema, and internally proxies to Anthropic / Gemini / Cohere / Ollama / anything. The client does not need to know which provider answered.

Model aliases

Instead of claude-3-5-sonnet-20251022, your code says our_smart_model. The gateway maps aliases to real models. When Anthropic ships Claude 4, you change the alias server-side and your code stays as it was.

Fallback chains

primary: openai/gpt-4o
on 5xx: anthropic/claude-3-5-sonnet
on 5xx: google/gemini-1.5-pro
on 5xx: refuse

Gateways define this in config. Retries count against a budget so a fallback cascade does not blow up your cost.

Semantic caching

Identical or near-identical prompts hit a cache instead of the provider. Savings on repeated agent loops can be 30 to 60 percent. Keys are embedding-based, so near-identical prompts share a cache slot.

Guardrails

Gateway-level:

  • PII redaction. Regex or ML-based pass before sending prompts.
  • Policy violations. Reject prompts with prohibited content.
  • Output filters. Scrub completions for leaks.

Portkey and Kong ship opinionated guardrails. LiteLLM leaves them optional.

Per-key rate limits

Give each team its own API key. Per-key budgets then stop one team from eating the shared quota, and most gateways support this.

Self-hosted vs managed trade-offs

FactorLiteLLM (self-hosted)OpenRouter (managed)Portkey (production)
CodeOpen source, PythonManaged SaaSOpen source (Mar 2026) + managed
SetupDeploy a proxySign upEither
Providers100+300+100+
BillingYour own keysOpenRouter creditsYour own keys
ObservabilityOpenTelemetryDashboardFull OTel + PII redaction
Best forTeams that want full controlRapid prototypingProduction with compliance

LiteLLM wins when you have an SRE team and want data sovereignty. OpenRouter wins when you want a single subscription and no infrastructure. Portkey wins when you need guardrails and compliance out of the box.

Cost tracking

A request carries provider, model, input_tokens, and output_tokens. Multiply those by per-model, per-token prices from a pricing sheet the gateway maintains, then aggregate per user, per team, or per project.

MCP plus routing

A gateway can route both LLM calls and MCP sampling requests. When a sampling request's modelPreferences prefer a specific model, the gateway translates that to the right backend. This is where the MCP gateway from Phase 13 · 17 and a routing gateway sometimes merge into one service.

Routing strategies

  • Static priority. First in list; fall back on error.
  • Load balancing. Round-robin or weighted.
  • Cost-aware. Pick the cheapest model meeting latency / quality.
  • Latency-aware. Pick the fastest model in the last N minutes.
  • Task-aware. Prompt classifier routes coding to one model, summarization to another.
Check: The Concept0/19
1. What does it mean for a gateway to be OpenAI-compatible?Define
A card with a chat request body holding a model field and a messages list with one user message, an arrow into a box named OpenAI-compatible gateway, and a dashed question box asking what it does with the request.
Typed example with illustrative names, not captured output.
2. What is a model alias used for?Define
A code card with two calls, route with the name smart and route with the name fast, an arrow to a dashed question box asking about the role of the name smart.
Call shapes taken from main.py; typed.
3. What is a fallback chain?Define
A request box with an arrow to a box for openai/gpt-4o that answers with a 503 status, and a dashed question box labelled fallback chain.
Typed schematic, not captured output.
4. Which technique saves 30 to 60 percent on repeated agent loops?Define
Three turns of an agent loop, each sending a similar prompt about open tickets to one provider box that shows three calls, and a dashed question box about what saves 30 to 60 percent.
Prompts are illustrative; typed.
5. Which three gateway-level guardrails does the lesson list?Define
A box for your app with an arrow into a gateway box and on to an LLM box, and a dashed question box labelled gateway-level guardrails.
Typed schematic, not captured output.
6. What does a per-key rate limit accomplish?Define
A code card for one API key with a key name and a limit field set to a question mark, an arrow to a dashed question box asking what it accomplishes.
Typed example with illustrative names, not captured output.
7. How does the gateway compute per-request cost?Define
A code card of one request record with provider openai, model gpt-4o, input tokens 1200 and output tokens 300, an arrow to a dashed question box asking for its cost.
Token counts are illustrative; typed.
8. What is LiteLLM?Define
A card named LiteLLM listing 100+ providers and OpenTelemetry observability, with an arrow to a dashed question box asking what it is.
Rows copied from the lesson's comparison table; typed.
9. What is OpenRouter?Define
A card named OpenRouter listing 300+ providers, with an arrow to a dashed question box asking what it is.
Rows copied from the lesson's comparison table; typed.
10. What is Portkey?Define
A card named Portkey listing 100+ providers, with an arrow to a dashed question box asking what it is.
Rows copied from the lesson's comparison table; typed.
11. Which statement correctly separates LiteLLM from OpenRouter on who pays the providers?Explain
Two rows, each with your app, a gateway and a provider box; the first gateway is LiteLLM and the second is OpenRouter, and each row ends in a dashed box asking who pays the provider.
Typed schematic, not captured output.
12. Why does a cost record carry the model name along with the token counts?Explain
A code card of a usage record with provider anthropic, model claude-sonnet and input and output token counts, the model line outlined with dashes, and a dashed question box asking why model is in the record.
Token counts are illustrative; typed.
13. How does task-aware routing differ from cost-aware routing?Explain
A card with one incoming prompt, a write a unit test request, above two boxes named task-aware routing and cost-aware routing, and a dashed question box below them asking how each picks a model.
Prompt is illustrative; typed.
14. Per the comparison table, which option wins for teams with an SRE team that want data sovereignty?Explain
A dashed boundary labelled the company holding an SRE team box and a your data box, a line reading requirement data sovereignty, and a dashed question box asking which gateway fits.
Typed schematic, not captured output.
15. When do a routing gateway and an MCP gateway sometimes merge into one service?Explain
A routing gateway box and an MCP gateway box side by side, and a dashed question box asking when they merge.
Typed schematic, not captured output.
16. Why do gateways count fallback retries against a budget?Explain
A request box followed by three provider boxes from the smart chain in main.py, each outlined in red and numbered attempt 1 to 3, and a dashed question box asking why retries count against a budget.
Chain names from main.py; the failures are drawn, not run.
17. A gateway with a six-provider chain tries every provider for each request during a long outage, and spend climbs. Which ingredient from the Concept section would cap this?Analyze
A request box followed by six provider boxes p1 to p6 in a chain, all outlined in red, with lines saying every provider is tried for each request and spend climbs, and a dashed question box asking what would cap this.
Typed schematic, not captured output.
18. Team B's requests start failing because Team A's nightly batch job used up the shared quota. Every team currently uses one API key. What fixes this?Analyze
Team A running a nightly batch and Team B with failing requests both sending through the same key to a gateway, whose shared provider quota bar is full, and a dashed question box asking what fixes it.
Typed schematic, not captured output.
19. A 4-person startup has no ops capacity and wants one invoice for many providers. Which option fits best?Design
A card describing a startup with a team size of 4, no ops capacity, and a wish for one invoice for many providers, with an arrow to a dashed question box asking for the best fit.
Typed from the stem.

Use It

code/main.py is a routing gateway in about 150 lines. It accepts OpenAI-shaped requests, dispatches to per-provider stubs, runs a priority fallback chain, tracks per-request cost, and redacts PII from the input before dispatch. Run it and you get four scenarios: a normal request, a primary-provider outage that triggers fallback, PII caught by redaction, and every provider in a chain down at once.

What to look at:

  • ROUTES maps an alias to a priority-ordered list of concrete providers. An alias that is not in the dict is treated as a concrete model name.
  • The fallback loop catches RuntimeError and moves to the next provider. The stub raises it to simulate a 5xx, and the loop does not inspect the status, so any RuntimeError triggers a retry.
  • The cost tracker multiplies token usage by per-model rates. The rates in PRICES are fake demo numbers, and token counts are estimated from string length.
  • The PII redactor scrubs SSN-shaped and 16-digit card-shaped patterns from the messages before forwarding.
Check: Use It0/7
1. In main.py, openai/gpt-4o and anthropic/claude-sonnet are both in OUTAGE and route("smart", ...) is called. What results?Apply
A code card showing a set of two models in outage, the smart chain of three models from main.py, and the route call, with a dashed question box asking what inv holds after the call.
Code lines copied from main.py; typed.
2. In main.py scenario 1, route("smart", ...) gets the 11-character message "explain MCP" and openai/gpt-4o is up. The stub reports 11 // 4 = 2 prompt tokens and max(20, 11 // 3) = 20 completion tokens. What is cost_usd?Apply
A card showing the message explain MCP with 11 characters and the stub token counts 2 and 20, a card with the gpt-4o and Sonnet rates from main.py, and a dashed question box asking for cost_usd.
Rates copied from main.py (abridged); typed.
3. main.py calls route("gpt-9-preview", ...), a name that is not a key in ROUTES, and the stub succeeds. What do chosen_model and cost_usd show?Apply
A code card listing the ROUTES keys smart and fast and a route call with the name gpt-9-preview, and dashed question boxes for chosen_model and cost_usd.
Call typed; ROUTES keys from main.py.
4. main.py routes route("fast", ...) with the message "my card is 4111111111111111 thanks". What happens?Apply
A code card with a route call on the fast alias whose user message contains a 16 digit card number, and a dashed question box asking what the call does with the message.
Call typed.
5. In main.py scenario 4, both models in the "fast" chain are in OUTAGE. What does the returned Invocation hold?Apply
A code card putting both models of the fast chain in outage and calling route, a card showing the Invocation dataclass fields, and a dashed question box asking what response, error and attempts hold.
Code typed from main.py.
6. Your gateway fails over from one provider to the next on any RuntimeError, and a user sends a malformed request that returns a 400. What happens, and what should change?Analyze
The fallback loop from route() that catches RuntimeError and continues, then a user with a malformed request, a provider box that answers 400, and a dashed question box asking what happens.
Loop lines copied from main.py.
7. A user's message contains the card number 4111-1111-1111-1111, and it reaches the provider unredacted. What explains this?Analyze
A user message containing a card number written in four dashed groups of four digits, an arrow to a provider box, a line showing the provider received the same text, and a dashed question box asking why it was not redacted.
Text is the stem's example; typed.

Ship It

This lesson produces outputs/skill-routing-config-designer.md. Give it a workload profile (latency, cost, compliance) and it picks LiteLLM, OpenRouter, or Portkey and produces a routing config.

Author's Note

I'd run main.py before reading the concepts again. Scenario 2 marks openai/gpt-4o as down and the attempts list shows the chain walking to anthropic/claude-sonnet, with the cost computed at Sonnet's rate instead of GPT-4o's. Seeing that once makes fallback chains and cost attribution concrete.

The part I'd read twice is the except RuntimeError branch in route(). It retries on any RuntimeError, and the demo only ever raises one for a simulated outage, so it behaves like "retry on 5xx" here. A real gateway has to separate a provider outage from a bad request, because failing over on a 400 just sends the same bad prompt to three providers. The retry budget from the Concept section is also missing from this code. The chain is short enough that it doesn't matter in the demo, but it would matter with a longer chain and real prices.

My bet is that you'll end up picking between LiteLLM, OpenRouter, and Portkey on operational grounds (who runs the proxy, whose keys pay, how much compliance you need) more than on routing features. The toy gateway shows the routing part is small. The redactor here is two regexes, which is a demo and nothing you'd ship as your only PII control.

Exercises

  1. Run code/main.py. Trigger the outage scenario and confirm the fallback lands on the second provider and the cost is attributed to it.

  2. Add semantic caching: SHA256 of the prompt is a lookup key; cache hits return instantly. Measure cost savings on a repeated call.

  3. Add a prompt classifier that routes "code ..." prompts to an alias favoring intelligence and "summarize ..." prompts to an alias favoring speed.

  4. Design per-team budgets: each team has a monthly spend cap; gateway refuses requests once cap is hit. Pick an enforcement granularity (per-request or windowed).

  5. Read LiteLLM, OpenRouter, and Portkey docs side by side. Name the one feature each ships that the other two do not.

Key Terms

TermWhat people sayWhat it actually means
Routing gateway"LLM proxy"One-API-surface layer in front of many providers
OpenAI-compatible"Speaks the OpenAI schema"Accepts /v1/chat/completions shape, translates to any backend
Model alias"our_smart_model"Name in your code that the gateway maps to a concrete model
Fallback chain"Retry list"Ordered list of providers attempted on failure
Semantic caching"Prompt-embedding cache"Key is embedding of the prompt; near-duplicates share a cache hit
Guardrails"Input/output filters"Redact PII, reject policy violations
Per-key rate limit"Team budget"Quota scoped to an API key
Cost tracking"Per-request spend"Aggregate token usage x price per model
LiteLLM"The open proxy"Self-hostable OSS routing gateway
OpenRouter"The managed SaaS"Hosted gateway with credit-based billing
Portkey"The production option"Open-source + managed with guardrails built in

Further Reading

Code

Ships With