Multi-Region LLM Serving and KV Cache Locality

Your service runs in us-east-1, us-west-2, and eu-west-1. You put an ALB in front with round-robin. Prefix cache hit rate in production drops to 8% and TTFT P50 triples. Your vLLM logs show requests paying full prefill cost almost across the board.

Round-robin is optimal for stateless services. LLM inference is stateful by design, because the KV cache encodes what the model has already seen. Routing blind means routing into the wrong cache.

Separately, your team has a DR plan. You back up model weights to S3 cross-region. A regional outage hits, you attempt failover, and the replica refuses to start. The tokenizer.json, the quantization config, and the RoPE scaling config lived in a separate bucket you didn't sync.

Multi-region LLM serving is a cache problem, a routing problem, and a DR-hygiene problem. The load balancer is the smallest part of it.

Cache-aware routing

A request arrives with a prompt. The router hashes the prefix (say, the first 512 tokens) and asks each replica whether it has that prefix cached. Replicas publish KV-cache events on a pub/sub channel as they allocate and evict blocks. The router picks the replica with the match and falls through to a GPU-util-based tie-breaker if none has it.

vLLM Router (Rust, 2026 production-stack) subscribes to kv.cache.block_added events, maintains a prefix-hash → replica index, and routes with an O(1) lookup. It falls through to least-queue-depth when nothing matches.

llm-d router follows the same pattern but is Kubernetes-native. It publishes events via the ControlPlane API.

SGLang RadixAttention (Phase 17 · 06) is the intra-replica equivalent. Cross-replica routing sits upstream of it.

Numbers

TTFT P50 on a 2K-token prompt, Llama 3.3 70B FP8, H100:

• Cache hit (same replica, prefix resident): ~80 ms.
• Cache miss (cold prefill): ~800 ms.

That is a 10x gap. If your router hits 60-80% of prefix cache across replicas, you approximate single-replica performance at N-replica capacity. At 10% you approximate naive scaling.

Cross-region adds a constraint: network latency

Inter-region RTT:

• us-east-1 ↔ us-west-2: ~65 ms.
• us-east-1 ↔ eu-west-1: ~75 ms.
• us-east-1 ↔ ap-southeast-1: ~220 ms.

Say routing sends a request from us-east-1 to a hot prefix in ap-southeast-1. On a 2K-token prompt the hit still wins: 80 ms plus a 220 ms RTT is 300 ms, against 800 ms for a cold prefill. On a short prompt where a cold prefill costs 250 ms, the 170 ms saved does not cover the round trip, and the remote hit loses. GORGO (2026 research) makes this explicit: minimize prefill_time + network_latency jointly, not prefill alone. Often the answer is to keep routing regional, and cross regions only for very long prefixes where prefill dominates.

Commercial "cross-region inference" does not help here

AWS Bedrock cross-region inference routes requests to other regions during capacity pressure. It optimizes availability, not TTFT, and treats inference as opaque. GKE Multi-Cluster Gateway does the same at the service level: failover with no awareness of the KV cache.

You still want an app-layer cache-aware router on top of these. They cover the "us-east-1 is on fire" case, and cache-aware routing covers the TTFT case.

DR hygiene: the 32% missing-files problem

A widely cited 2026 stat says 32% of LLM DR failures happen because teams backed up weights but forgot:

• tokenizer.json or tokenizer.model
• Quantization configs (quantize_config.json, AWQ scales, GPTQ zero-points)
• Model-specific configs (RoPE scaling, attention masks, chat templates)
• Engine config (vllm_config.yaml, sampling defaults, LoRA adapter manifests)

The fix is a three-file minimum DR manifest:

1. All files under the HF model repo (weights + configs + tokenizer).
2. Engine-specific serving config.
3. Deployment manifest (K8s YAML, Dockerfile, dependency lock).

Then run a DR drill quarterly. The JPMorgan us-east-1 failover in Nov 2024 recovered in 22 minutes against a 30-minute SLA, and a rehearsed playbook is what makes a number like that possible.

Data residency is orthogonal

EU customer PHI is not allowed to leave the EU. If your cache-aware router sends a Paris-originated request to us-east-1 for a prefix match, you have violated GDPR regardless of the TTFT gain. Partition routers by residency boundary before you optimize for cache.

Numbers to remember

• Cache hit vs miss TTFT gap: ~10x (80 ms vs 800 ms on a 2K prompt).
• Inter-region RTT US-EU: ~75 ms.
• DR failure: 32% miss tokenizer/quant configs.
• JPMorgan us-east-1 failover, Nov 2024: 22 minutes (30-min SLA).

code/main.py simulates three routing strategies (round-robin, cache-aware regional, cache-aware global) on a multi-region workload. It reports cache hit rate, TTFT P50/P99, and the cross-region bill.

This lesson produces outputs/skill-multi-region-router.md. Given regions, residency constraints, and an SLA, it designs a routing plan.

1. Run code/main.py, then lower CACHE_MISS_MS (a stand-in for shorter prompts) until the GLOBAL strategy stops beating REGIONAL on mean TTFT. At what miss cost does cross-region routing stop paying off, given the RTT table?
2. Your cache hit rate drops from 70% to 12%. Diagnose three possible causes and the observables that would confirm each.
3. Design a DR manifest for a 70B AWQ-quantized model served in vLLM with 5 LoRA adapters. List every file and config.
4. Argue whether Bedrock cross-region inference is "enough" for a fintech with strict TTFT SLOs. Cite specific behaviors.
5. A Paris-origin request matches a prefix in us-east-1. Do you route it? Write the policy.

• BentoML: Multi-cloud and cross-region inference
• arXiv: GORGO (2602.11688): cross-region KV-cache reuse with network latency term.
• TianPan: Multi-Region LLM Serving Cache Locality
• AWS Bedrock Cross-Region Inference: availability failover documentation.
• vLLM Production Stack Router: cache-aware router source.