TAI BUI
Back to curriculum
LearnPython

AI Gateways — LiteLLM, Portkey, Kong AI Gateway, Bifrost

A gateway sits between your apps and your model providers and handles routing, fallback, retries, rate limits, secret references, observability, and guardrails behind one API. The 2026 market splits roughly by scale and posture. **LiteLLM** is MIT-licensed, covers 100+ providers, speaks the OpenAI API, and in Kong's benchmark breaks down around 2000 RPS (8 GB of memory, cascading failures), so it fits Python shops under 500 RPS and dev work. **Portkey** positions itself as a control plane with guardrails, PII redaction, jailbreak detection, and audit trails. It went Apache 2.0 in March 2026, adds 20-40 ms of latency, and charges $49/mo for the production tier. **Kong AI Gateway** builds on Kong Gateway. Kong's own benchmark on the same 12 CPUs has it 228% faster than Portkey and 859% faster than LiteLLM, at $100/model/month (5 models max on the Plus tier), so it suits teams already on Kong. **Bifrost** (Maxim AI) retries with configurable backoff and can fall back to Anthropic when OpenAI returns a 429. **Cloudflare and Vercel AI Gateways** are managed, need no ops, and keep to basic retry. Data residency usually decides self-hosted versus managed, and Portkey and Kong sit in the middle with open-source and optional managed offerings.

TIME: ~60 minutesPREREQUISITES: Phase 17 · 01 (Managed LLM Platforms), Phase 17 · 16 (Model Routing)

Learning Objectives

  • Enumerate the seven core gateway features (routing, fallback, retries, rate limits, secrets, observability, guardrails).
  • Map four 2026 gateways (LiteLLM, Portkey, Kong AI, Bifrost) to scale ceilings and use cases.
  • Cite the Kong benchmark (228% vs Portkey, 859% vs LiteLLM) and explain why it matters for >500 RPS.
  • Choose self-hosted vs managed given data residency and ops budget.

The Problem

Your product calls OpenAI, Anthropic, and a self-hosted Llama. Each has its own SDK, error model, rate limit, and auth scheme. You want failover (if OpenAI returns a 429, try Anthropic), one credential store, one place to watch traffic, and rate limits per tenant.

Building that in the app couples every service to every provider. A gateway pulls it into one process with one API, usually OpenAI-compatible, that fans out to the providers behind it.

Warm-up0/3
1. Your app gets a 429 from OpenAI during a traffic spike. Based on model routing, what is the sensible response?Warm-up
A bar chart of usual traffic against a spike right now, with a 429 too many requests response coming back from the provider. A line asks what the sensible response is.
Setup: the traffic spike and the 429 response, nothing more. The sensible response is left open.
2. What does time to first token (TTFT) measure for an LLM request?Warm-up
A timeline from request sent to done, with span A marking the time before any token shows up and span B marking tokens arriving one by one afterward. A line asks which span is time to first token.
Setup: both spans on the timeline, unlabeled by name. Which one is TTFT is left open.
3. Where should an application keep its provider API keys in production?Warm-up
A service that needs a secret key to authenticate to a paid API, with the key's actual storage location left as an unlabeled dashed box. A line asks where that key should live in production.
Setup: the service and its need for a key, given. Where the key is stored is left open.
Check: The Problem0/2
1. What is an AI gateway?Define
Your product wired directly to three providers, OpenAI, Anthropic and a self-hosted Llama, each labeled with its own SDK, error model or auth scheme, plus a note that each also has its own rate limit. A line says an AI gateway is introduced to address this and asks what it is.
Setup: the direct, separate wiring to each provider and its own friction. What the gateway actually is stays open.
2. Why put one OpenAI-compatible endpoint in front of several model providers?Explain
A table of three providers, OpenAI, Anthropic and Self-hosted, each with its own SDK, error model, rate limit and auth column all marked own. A line asks why put one OpenAI-compatible endpoint in front of all three.
Setup: the full grid of separate integration work the given providers require. Why one endpoint fixes it is left open.

The Concept

Core features

  1. Provider routing: OpenAI, Anthropic, Gemini, self-hosted, and others behind one API.
  2. Fallback: on a 429, a 5xx, or a quality failure, retry somewhere else.
  3. Retries: exponential backoff with a bounded number of attempts.
  4. Rate limits: per tenant, per key, per model.
  5. Secret references: pull credentials from a vault at runtime so the app code holds none.
  6. Observability: OTel with GenAI attributes (Phase 17 · 13) plus cost attribution.
  7. Guardrails: PII redaction, jailbreak detection, allowed-topic filters.

LiteLLM: MIT OSS, Python

  • 100+ providers, OpenAI-compatible, router config, fallback, basic observability.
  • Breaks down around 2000 RPS in Kong's benchmark, with an 8 GB memory footprint and cascading failures under sustained load.
  • Best fit: a Python app under 500 RPS, dev and staging gateways, experimental routing.
  • Cost: $0 for OSS, and the cloud version has a free tier.

Portkey: the control-plane pitch

  • Apache 2.0 OSS as of March 2026. Guardrails, PII redaction, jailbreak detection, audit trails.
  • 20-40 ms of latency overhead per request.
  • $49/mo for the production tier, which adds retention and an SLA.
  • Best fit: regulated industries that want guardrails and observability bundled.

Kong AI Gateway: the scale play

  • Built on Kong Gateway, a mature API gateway product (Lua on OpenResty).
  • Kong's own benchmark on a 12-CPU equivalent has it 228% faster than Portkey and 859% faster than LiteLLM. Read that as a vendor number, since Kong ran it.
  • Pricing: $100/model/month, 5 models max on the Plus tier.
  • Best fit: teams already on Kong, past 1000 RPS, willing to pay for a license.

Bifrost (Maxim AI)

  • Automatic retries with configurable backoff.
  • Falling back to Anthropic on an OpenAI 429 is the canonical recipe.
  • A newer, commercial entrant.

Cloudflare AI Gateway and Vercel AI Gateway

  • Managed, zero-ops, with basic retry and observability.
  • Best fit: JavaScript apps served from the edge on Cloudflare or Vercel.
  • Thinner than Kong or Portkey on guardrails and rate limits.

Self-hosted vs managed

Data residency forces the decision. Healthcare and finance default to self-hosting (LiteLLM, Portkey OSS, or Kong). Consumer products default to managed (Cloudflare AI Gateway) or a middle tier (Portkey managed). A hybrid works too: self-host for the regulated tenant and use managed for everyone else.

Latency budget

  • LiteLLM: 5-15 ms overhead, typical.
  • Portkey: 20-40 ms.
  • Kong: 3-8 ms.
  • Cloudflare and Vercel: 1-3 ms, thanks to the edge.

Gateway latency adds straight onto TTFT. For a TTFT P99 under 100 ms, pick Kong or Cloudflare. Under 500 ms, any of them fits.

Rate-limit semantics matter

A simple token bucket holds up at moderate scale. Multi-tenant traffic needs a sliding window, a burst allowance, and per-tenant tiering. LiteLLM ships a token bucket, Kong ships a sliding window, and Portkey ships tiered limits.

Gateway, observability, and routing compose

Phase 17 · 13 (observability), 16 (model routing), and 19 (gateways) describe one layer in production. Either pick a tool that covers all three or wire them together with care. Most 2026 deployments split the roles, pairing Helicone (observability) or Portkey (guardrails) with Kong (scale).

Numbers to remember

  • LiteLLM: breaks at ~2000 RPS, 8 GB memory.
  • Portkey: 20-40 ms overhead, Apache 2.0 since March 2026.
  • Kong: 228% faster than Portkey, 859% faster than LiteLLM.
  • Kong pricing: $100/model/month, 5 max on the Plus tier.
  • Cloudflare and Vercel: 1-3 ms overhead at the edge.
Check: The Concept0/27
1. What is Cloudflare AI Gateway?Define
A card naming Cloudflare AI Gateway and listing it among LiteLLM, Portkey, Kong AI Gateway and Bifrost. A line asks what it is.
Setup: the name, in the company of the other named gateways. What it is is left open.
2. What does PII redaction do inside a gateway?Define
A card naming PII redaction and listing it among three guardrail mechanisms, alongside jailbreak detection and an allowed-topic filter. A line asks what it does.
Setup: the term, among its sibling guardrail mechanisms. What it does is left open.
3. What is jailbreak detection in a gateway?Define
A card naming jailbreak detection and listing it among the three guardrail mechanisms, alongside PII redaction and an allowed-topic filter. A line asks what it is.
Setup: the term, among its sibling guardrail mechanisms. What it is is left open.
4. What is an audit trail in an AI gateway?Define
A card naming audit trail and placing it in the lesson's compliance discussion, kept distinct from guardrails, PII redaction and jailbreak detection. A line asks what it is.
Setup: the term, set apart from its neighboring compliance concepts. What it is is left open.
5. How does a token-bucket rate limiter decide whether to allow a request?Define
A card naming token-bucket as one of two rate-limiter types this lesson names, the other being sliding-window. A line asks how it decides whether to allow a request.
Setup: the name, set against its sibling limiter type. How it actually decides is left open.
6. What is a sliding-window rate limiter?Define
A card naming sliding-window as the second rate-limiter type the lesson names, set against token-bucket. A line asks what it is.
Setup: the name, set against its sibling limiter type. What it is is left open.
7. What scale ceiling does Kong's benchmark report for LiteLLM?Define
A card naming LiteLLM, MIT-licensed with 100+ providers, as one of four gateways this lesson compares. A line asks what scale ceiling Kong's benchmark reports for it.
Setup: the name and its license and provider count. The reported ceiling is left open.
8. On the same 12 CPUs, how much faster does Kong's own benchmark say Kong AI Gateway is?Define
A card naming Kong AI Gateway, built on Kong Gateway, benchmarked by Kong itself on the same 12 CPUs as the other two gateways. A line asks how much faster Kong says it is than Portkey and LiteLLM.
Setup: who ran the benchmark and on what hardware. The reported speedup figures are left open.
9. Which gateway adds 20-40 ms of overhead and focuses on guardrails, PII redaction, and jailbreak detection?Define
A card listing a gateway's two given traits, 20 to 40 milliseconds of overhead and a focus on guardrails, PII redaction and jailbreak detection, with no gateway named. A line asks which gateway this is.
Setup: only the two given traits from the stem. Which named gateway they describe is left open.
10. How does Bifrost (Maxim AI) handle an OpenAI 429?Define
A card naming Bifrost from Maxim AI and listing it among LiteLLM, Portkey, Kong AI Gateway, Cloudflare and Vercel. A line asks how it handles an OpenAI 429.
Setup: the name, among the other named 2026 gateways. How it handles a 429 is left open.
11. A multi-tenant SaaS needs per-tenant limits for free, trial, and paid tiers. How do the gateway defaults differ?Define
Three gateways, LiteLLM, Kong and Portkey, each with its own rate-limit default left blank, in the context of serving free, trial and paid tenants differently. A line asks how the defaults differ.
Setup: the three named gateways, each needing a default filled in. How they actually differ is left open.
12. What changed for Portkey in March 2026, and what does the production tier cost?Define
A card naming Portkey as one of four 2026 gateways this lesson compares, alongside LiteLLM, Kong AI Gateway and Bifrost. A line asks what changed for it in March 2026 and what the production tier costs.
Setup: the name, among the other named gateways. What changed and what it costs are both left open.
13. Why does a gateway fall back to a different provider on a 429 or 5xx instead of only retrying the same one?Explain
Two providers, A and B, each noted only as something that may fail, with no rates or outcomes shown. A line says a chain tries both before giving up, instead of retrying A alone, and asks why that fails less often.
Setup: two providers that may each fail, nothing quantified. Why chaining them fails less often is left open.
14. Why does the lesson tell you to read Kong's 228% and 859% figures as a vendor number?Explain
A benchmark's design and run flowing into Kong AI Gateway, the very gateway being tested, with a note that the same company designed it, ran it, and makes the winning gateway. A line asks why to read the result as a vendor number.
Setup: who designed and ran the test, and that it is their own gateway being tested. Why that matters for reading the result is left open.
15. Why does Kong's throughput advantage matter mainly for traffic above about 500 RPS?Explain
A traffic axis with a line marked at 500 RPS, and a shaded region below it noted as where a Python gateway already fits fine. A line asks why a faster gateway's throughput gap matters mainly above that line.
Setup: only the 500 RPS line and the note about what fits below it. Why the gap matters mainly above it is left open.
16. Why does gateway overhead matter when you have a tight TTFT target?Explain
A request's wait split into the provider's own time and an added gateway overhead, stacked one after the other before the first token arrives, with no specific amounts given. A line asks why that matters for a tight TTFT target.
Setup: that the two amounts stack, with neither quantified. Why that matters for a tight target is left open.
17. What does fallback buy, and what does it cost?Explain
A request where provider A fails and provider B then succeeds, taking longer than going straight to B, with a note comparing this to the request simply failing outright. A line asks what the trade gets you and what it spends.
Setup: the general fallback mechanism, not resolved into named gains or costs. What is bought and what is spent are both left open.
18. Why does a gateway pull provider credentials from a vault at runtime?Explain
App code holding no keys, a separate vault, and an arrow from the vault to the outgoing provider call, labeled as fetched when the request happens. A line asks why the gateway reaches into a vault for this instead of the app holding the key.
Setup: that app code holds nothing and the vault supplies the key at request time. Why that arrangement matters is left open.
19. What usually decides between a self-hosted and a managed gateway?Explain
A card naming the self-hosted versus managed tradeoff as something the lesson works through for every gateway it covers. A line asks what usually decides it.
Setup: the tradeoff named, nothing about what drives the decision. What usually decides it is left open.
20. Your SLA is TTFT P99 under 100 ms. Which gateways fit the overhead budget?Apply
A card stating a TTFT P99 budget of under 100 milliseconds and the fact that gateway overhead adds straight onto the provider's own time. A line asks which gateways fit this budget.
Setup: only the budget number and that overhead adds on top. No per-gateway figures are shown; which ones fit is left open.
21. Your LiteLLM gateway sits at 2500 sustained RPS with memory near 8 GB, and failures are cascading. What explains the symptoms?Analyze
A card listing a LiteLLM gateway's current symptoms: 2500 sustained RPS, memory near 8 GB, and cascading failures. A line asks what explains them.
Setup: only the given symptom numbers, no diagnosis. What explains them is left open.
22. After one provider starts degrading, a gateway's success rate stays near 100% but mean latency climbs. Is something broken?Analyze
Two side-by-side trend lines over time as one provider degrades: success rate staying nearly flat near the top, and mean latency climbing. A line asks whether something is broken.
Setup: both given trends, flat success and rising latency, with no verdict drawn. Whether something is broken is left open.
23. A team on Cloudflare AI Gateway wants per-tenant tiered limits and deep guardrails. What assumption failed?Analyze
A card listing a team's two wants on Cloudflare AI Gateway, per-tenant tiered rate limits and deep guardrails. A line asks what assumption failed.
Setup: only the two given wants, no verdict. What assumption failed is left open.
24. A healthcare customer needs self-hosting, PII redaction, and audit trails. Which pick fits best?Design
A card listing a healthcare customer's three needs: self-hosting, PII redaction, and an audit trail. A line asks which pick fits best.
Setup: only the three given needs. Which pick fits best is left open.
25. Most of your customers are consumers, but one hospital tenant requires its data to stay in infrastructure you control. Which setup fits?Design
A card describing a mostly consumer customer base with one hospital tenant whose data must stay in infrastructure the team controls. A line asks which setup fits.
Setup: only the given customer mix. Which setup fits is left open.
26. Your team already runs Kong past 1000 RPS and now needs bundled guardrails and observability. What does the lesson suggest?Design
A card describing a team already running Kong past 1000 RPS that now also needs bundled guardrails and observability. A line asks what the lesson suggests.
Setup: only the given facts about the team's current setup and new need. What the lesson suggests is left open.
27. A one-person team ships a JavaScript app from Cloudflare's edge, handles no regulated data, and wants TTFT P99 under 100 ms. Which gateway fits?Design
A card describing a one-person team shipping a JavaScript app from Cloudflare's edge, with no regulated data and a TTFT P99 target under 100 milliseconds. A line asks which gateway fits.
Setup: only the given team facts and latency target. Which gateway fits is left open.

Use It

code/main.py simulates gateway routing with fallback across 3 providers under 429/5xx injection. It reports latency, retry rate, and fallback hit rate.

Check: Use It0/8
1. main.py chains OpenAI (3% error), Anthropic (2%), and Self-hosted (5%). If errors are independent, what share of requests fail all three?Apply
A table of main.py's three chained providers and their given error rates, OpenAI 3 percent, Anthropic 2 percent, Self-hosted 5 percent, not yet combined. A line asks what share of requests fail all three if errors are independent.
Setup: the three given rates from the stem, not yet multiplied. The combined failure share is left open.
2. In main.py, a request goes through the Kong profile. OpenAI errors and Anthropic succeeds. What latency does the request record?Apply
An excerpt of main.py's own constants: Kong's gateway overhead of 5, OpenAI's base latency of 180, Anthropic's base latency of 220, and the comment showing a failed call is charged at 30 percent of its base. A line says this request has OpenAI error and Anthropic succeed, and asks the total latency recorded.
Setup: the real constants from main.py, not yet combined for this specific request. The total latency is left open.
3. In main.ts, the free tenant has capacity 2 and refills 0.5 tokens per second. It sends four requests back to back. What does the gateway return?Apply
A token bucket for the free tier starting at 2 tokens, refilling at 0.5 tokens per second, with four requests shown arriving back to back. A line asks what the gateway returns for each.
Setup: the bucket's starting capacity and refill rate, and the four requests with no gap between them. What each one gets back is left open.
4. In main.ts, the paid tenant's first request reaches the openai mock, which returns a retryable 429 on its first call. With maxAttempts 2, which provider answers and how many fallback hits are recorded?Apply
The given behavior for the paid tenant's openai mock, a 429 on its first call and success afterward, with a note that the fallback chain is openai then anthropic then self-hosted. A line asks which provider answers and how many fallback hits are recorded.
Setup: the mock's own given call-by-call behavior and the chain order. Which provider answers and the fallback hit count are both left open.
5. For that same paid request in main.ts (baseBackoffMs 1, jitter 1.0, overhead 5 ms, openai latency 180 ms), what totalLatencyMs is returned?Apply
A table of the given config for this request: baseBackoffMs 1, a fixed jitter of 1.0, gateway overheadMs 5, and the openai mock's 180 millisecond latency on its successful call. A line asks what totalLatencyMs is returned.
Setup: the four given config values, not yet combined. The returned total latency is left open.
6. In main.ts, a caller sends the key "nope", which no tenant has registered. What happens?Apply
The three stages of handle() in order: authenticate, rate limit, provider chain. A line says a caller sends the key nope, which no tenant has registered, and asks what happens.
Setup: the stage order and the given unregistered key. What actually happens is left open.
7. A teammate runs main.py, sees Cloudflare with the lowest mean latency, and concludes Cloudflare is the fastest gateway. What is wrong with that conclusion?Analyze
The lesson's own real terminal output: LiteLLM, Portkey, Kong and Cloudflare rows with their real success rate, mean latency, retries and fallback counts, where Cloudflare's mean latency is the lowest. A line says a teammate calls Cloudflare the fastest gateway from this and asks what's wrong with that.
Setup: the real simulator output. What's wrong with trusting its latency ranking is left open.
8. In main.ts, a free-tier tenant gets 429 rate limit exceeded while all three providers are healthy. Which component rejected the request?Analyze
The three stages of handle() in order, authenticate, rate limit, provider chain. A line says a free-tier tenant gets 429 while all three providers are healthy, and asks which component rejected it.
Setup: the stage order and the given fact that every provider is healthy. Which stage actually rejected the request is left open.

Ship It

The output here is outputs/skill-gateway-picker.md. Give it your scale, ops posture, compliance needs, and latency budget, and it picks a gateway.

Author's Note

I'd run main.py first and read the table, then the three lines under it. The table shows 100.0% success for every gateway, which is what a 3-provider chain should give over 1000 requests with a fixed seed, so the numbers repeat from run to run. The three lines under it are printed text, not measured output. They give the closed-form success rate for one provider (97%), two in a chain (99.94%, the complement of 0.03 × 0.02), and three (99.997%), and you can check them by hand. The chain in the simulation is OpenAI, then Anthropic, then self-hosted.

My bet is that the latency column is the one worth staring at. Each gateway only adds a fixed overhead (2 ms for Cloudflare up to 30 ms for Portkey) on top of whatever the providers cost, and a failed call is charged at 30% of that provider's base latency. Fallback buys success rate and pays for it in mean latency, which is the trade the Latency budget section is about.

Keep the toy in proportion, though. The overhead numbers are the lesson's typical figures typed in as constants, so the simulator cannot tell you which gateway is faster. The TypeScript port is closer to a real gateway: it checks the API key, applies a per-tenant token bucket by tier, retries with backoff, and walks the fallback chain. If you care about the rate-limit question in the exercises, read that bucket before you pick an algorithm.

Exercises

  1. Run code/main.py. Configure fallback from OpenAI→Anthropic→self-hosted. What's the expected hit rate at 5% provider error rate?
  2. Your SLA is TTFT P99 < 200 ms on a 300 ms baseline. Which gateways stay within budget?
  3. A healthcare customer requires self-hosted + PII redaction + audit. Pick Portkey OSS or Kong.
  4. Compare LiteLLM vs Kong: at what RPS ceiling should a team migrate?
  5. Design a rate-limit policy for a multi-tenant SaaS: free tier, trial tier, paid tier. Token-bucket or sliding-window?

Key Terms

TermWhat people sayWhat it actually means
Gateway"API broker"Process sitting between apps and providers
LiteLLM"the MIT one"Python OSS, 100+ providers, breaks at 2K RPS
Portkey"guardrails gateway"Control plane + observability, Apache 2.0
Kong AI Gateway"the scale one"Built on Kong Gateway, benchmark leader
Bifrost"Maxim's gateway"Retries + Anthropic fallback recipe
Cloudflare AI Gateway"edge managed"Edge-deployed managed gateway, zero-ops
PII redaction"data scrub"Regex + NER mask before sending to model
Jailbreak detection"prompt injection guard"Classifier on user input
Audit trail"regulated log"Immutable record of every LLM call
Token-bucket"simple rate limit"Refill-based rate limiter
Sliding-window"precise rate limit"Time-windowed rate limiter; better fairness

Further Reading

Code

Ships With