Self-Hosted Serving Selection — llama.cpp, Ollama, TGI, vLLM, SGLang
Four engines dominate self-hosted inference in 2026, and the pick comes down to hardware, scale, and ecosystem. **llama.cpp** is the fastest on CPU, with the widest model support and full control over quantization and threading. **Ollama** is the one-command install for a dev laptop. It runs roughly 15-30% slower than llama.cpp (Go, CGo, and HTTP serialization), and the gap grows to about 3x throughput under prod-like load. **TGI entered maintenance mode on December 11, 2025**, so it gets bug fixes only. It runs about 10% slower than vLLM on raw throughput, though it has historically had top observability and tight HF-ecosystem integration. That status makes it a risky long-term bet, and SGLang or vLLM are safer defaults for new projects. **vLLM** is the general-purpose production default: v0.15.1 (February 2026) adds PyTorch 2.10, RTX Blackwell SM120, and H200 optimization. **SGLang** is the specialist for agentic multi-turn and prefix-heavy work, with 400,000+ GPUs in production (xAI, LinkedIn, Cursor, Oracle, GCP, Azure, AWS). Hardware narrows the field fast. On CPU you are on llama.cpp. On AMD or other non-NVIDIA GPUs you are on vLLM (or SGLang), because TRT-LLM is NVIDIA-locked. The pipeline pattern for 2026: dev on Ollama, staging on llama.cpp, prod on vLLM or SGLang, with the same GGUF/HF weights throughout.
Learning Objectives
- Pick an engine given hardware (CPU / AMD / NVIDIA Hopper / Blackwell), scale (1 user / 100 / 10,000), and workload (general chat / agent / long-context).
- Name the 2026 TGI maintenance-mode status (December 11, 2025) and why it biases new projects toward vLLM or SGLang.
- Describe the dev/staging/prod pipeline using the same GGUF or HF weights throughout.
- Explain why "CPU only" forces llama.cpp and "AMD" excludes TRT-LLM.
The Problem
Your team starts a new self-hosted LLM project. One engineer says Ollama, another says vLLM, and a third asks whether TGI just works out of the box. Each of them is right in some context and wrong in others.
In 2026 the choice tree runs hardware first, scale second, workload third. One 2025 event also changes the default for new projects: TGI entered maintenance mode on December 11.
The Concept
The five engines
| Engine | Best for | Notes |
|---|---|---|
| llama.cpp | CPU / edge / minimal deps / widest model support | Fastest on CPU, full control |
| Ollama | Dev laptops, single user, one-command install | 15-30% slower than llama.cpp; 3x prod throughput gap |
| TGI | HF ecosystem, regulated industries | Maintenance mode Dec 11, 2025 |
| vLLM | General-purpose production, 100+ users | Broad production default; v0.15.1 Feb 2026 |
| SGLang | Agentic multi-turn, prefix-heavy workloads | 400,000+ GPUs in production |
Hardware-first decision
CPU only → llama.cpp. Ollama works too but is slower. I would not reach for anything else on CPU.
AMD GPU → vLLM (AMD ROCm support). SGLang also works. TRT-LLM is NVIDIA-locked, so it's off the table.
NVIDIA Hopper (H100 / H200) → vLLM, SGLang, or TRT-LLM. All three are top-tier here.
NVIDIA Blackwell (B200 / GB200) → TRT-LLM is the throughput leader (Phase 17 · 07). vLLM and SGLang follow close behind.
Apple Silicon (M-series) → llama.cpp (Metal). Ollama wraps this.
Scale-second decision
1 user / local dev → Ollama. One command, first-token in seconds.
10-100 users / small team → vLLM single-GPU.
100-10k users / production → vLLM production-stack (Phase 17 · 18) or SGLang.
10k+ users / enterprise → vLLM production-stack + disaggregated (Phase 17 · 17) + LMCache (Phase 17 · 18).
Workload-third decision
General chat / Q&A → vLLM wins on broad default.
Agentic multi-turn (tools, planning, memory) → SGLang's RadixAttention (Phase 17 · 06) does best.
RAG with heavy prefix reuse → SGLang.
Code generation → vLLM fine; SGLang slightly better on cache.
Long context (128K+) → vLLM + chunked prefill; SGLang + tiered KV.
The TGI maintenance trap
Hugging Face TGI entered maintenance mode on December 11, 2025, which means only bug fixes going forward. Historically it had top-tier observability and best-in-class HF-ecosystem integration (model cards, safety tools), and it ran slightly behind vLLM on raw throughput.
For new projects in 2026, default away from TGI. Existing TGI deployments can keep running, but plan a migration. SGLang and vLLM are the safer defaults.
The pipeline pattern
Dev (Ollama) → staging (llama.cpp) → prod (vLLM). Same GGUF or HF weights throughout. Engineers iterate quickly on laptops; staging mirrors production quantization; prod is the serving target.
Ollama caveat
Ollama is great for dev. It is a poor fit for shared production because Go HTTP serialization adds overhead, concurrency management is simpler than vLLM's, and OpenTelemetry support lags. Use Ollama where it shines (one user, one command) and switch to vLLM once the service is shared.
Self-hosted vs managed is a separate decision
Phase 17 · 01 (managed hyperscalers) and · 02 (inference platforms) cover managed. Here I assume you've already decided to self-host. Reasons to self-host: data residency, custom fine-tune, total cost ownership at scale, domain model not available on hosted.
Numbers you should remember
- TGI maintenance mode: December 11, 2025.
- vLLM v0.15.1: February 2026; PyTorch 2.10; Blackwell SM120 support.
- SGLang production footprint: 400,000+ GPUs.
- Ollama throughput gap vs llama.cpp: 15-30% slower; 3x under prod load.
Use It
code/main.py is a decision-tree walker. Give it hardware, scale, and workload, and it picks an engine and explains why.
Ship It
This lesson produces outputs/skill-engine-picker.md. Given constraints, it picks an engine and writes the migration plan.
I'd run main.py first and read the seven scenarios as a test of your own instincts. It is stdlib Python, and each scenario prints an engine plus a list of reasons, so you can see which branch fired and why.
The hardware branch comes first in the code, the same order the decision tree uses above. Scale and workload are then checked inside each branch, and scale also appends the enterprise stack note at the end. The workload check is a substring match on "agentic" or "prefix", so the long-context scenario on Hopper has no branch of its own and falls through to vLLM. If you care about long context, that is the first spot I'd extend, with the chunked-prefill and tiered-KV split from the workload section.
Two results might surprise you. A single user on CPU gets Ollama rather than bare llama.cpp, because the code treats Ollama as the one-command wrapper over it. Blackwell returns TRT-LLM, and the vLLM "close second" note only appears for small-team and production scale, so the enterprise Blackwell scenario never mentions it. The TGI maintenance-mode line gets appended to every result, which is the code's way of making that default hard to miss.
The code and the prose differ in two small places. The enterprise note in the code names a cache-aware router (Phase 17 · 11), where the scale section above names LMCache (Phase 17 · 18). And Apple Silicon with a single user returns Ollama, while the hardware section above lists llama.cpp as the engine underneath. Treat the code as a simplified walker and the prose as the fuller picture.
Exercises
- Run
code/main.pywith your hardware / scale / workload. Does the output match your intuition? - Your infra is 12 H100s and 8 MI300X AMD. What engine? Why is TRT-LLM off the table?
- A team wants to use TGI in 2026 because "it's what we know." Argue the migration case.
- Ollama dev to vLLM prod: what changes in quantization, configuration, and observability?
- RAG product with P99 prefix length 8K and high reuse across tenants. Pick an engine and stack it with Phase 17 · 11 + 18.
Key Terms
| Term | What people say | What it actually means |
|---|---|---|
| llama.cpp | "the CPU one" | Widest model support, fastest on CPU |
| Ollama | "the laptop one" | One-command install, dev-grade throughput |
| TGI | "HF's serving" | Maintenance mode since Dec 2025 |
| vLLM | "the default" | Broad production baseline 2026 |
| SGLang | "the agentic one" | Prefix-heavy, RadixAttention |
| TRT-LLM | "NVIDIA-locked" | Blackwell throughput leader, NVIDIA only |
| GGUF | "llama.cpp format" | Bundled K-quant variants |
| Production-stack | "vLLM K8s" | Phase 17 · 18 reference deployment |
| Pipeline pattern | "dev→stage→prod" | Ollama → llama.cpp → vLLM on same weights |
Code
- main.py (python, 84 lines)