LearnPython

GPU Autoscaling on Kubernetes — Karpenter, KAI Scheduler, Gang Scheduling

GPU autoscaling takes three layers. Karpenter provisions nodes dynamically in under a minute, reported as 40% faster than Cluster Autoscaler. KAI Scheduler adds gang scheduling, topology awareness, and hierarchical queues, which stops the 7-of-8 partial allocation trap where seven nodes sit waiting and billing on one missing GPU. Application-level autoscalers (NVIDIA Dynamo Planner, llm-d Workload Variant Autoscaler) scale on inference signals like queue depth and KV cache utilization instead of CPU or DCGM duty cycle. The classic HPA trap is that `DCGM_FI_DEV_GPU_UTIL` is a duty-cycle measurement, so 100% could be 10 requests or 100. vLLM pre-allocates KV cache memory, so memory-based scale-down doesn't trigger either. You'll compose the three layers and avoid Karpenter's default `WhenEmptyOrUnderutilized` policy, which can terminate running GPU jobs mid-inference.

TIME: ~75 minutesPREREQUISITES: Phase 17 · 02 (Inference Platform Economics), Phase 17 · 04 (vLLM Serving Internals)

Learning Objectives

Diagram the three autoscaling layers (node provisioning, gang scheduling, application-level) and name the tool used at each layer.
Explain why `DCGM_FI_DEV_GPU_UTIL` is the wrong HPA signal for vLLM and name two replacements (queue depth, KV cache utilization).
Describe gang scheduling and the partial-allocation failure mode KAI Scheduler prevents (7 of 8 GPUs idle).
Name the Karpenter consolidation policy (`WhenEmptyOrUnderutilized`) that terminates running GPU jobs and state the 2026 safe alternative.

The Problem

Your team ships an LLM-serving service on Kubernetes. You set up HPA with DCGM_FI_DEV_GPU_UTIL as the signal, and the service pins at 100% utilization during business hours. HPA doesn't scale up because it already thinks you're full. You add a replica by hand and TTFT drops, but HPA still sits there. The signal is lying to you.

Separately, you use Cluster Autoscaler for nodes. A 1M-token prompt arrives at 2 a.m., the cluster spends 3 minutes provisioning a node, and the request times out.

Separately again, you deploy a 70B model that needs 8 GPUs across 2 nodes. The cluster has 7 GPUs free and 1 missing, spread across 3 nodes. Cluster Autoscaler provisions a node for the missing GPU, and seven nodes wait 4 minutes, burning money, while Kubernetes brings up the last one.

Three layers, three different failure modes. GPU-aware autoscaling in 2026 takes more than turning on HPA: you compose node provisioning, gang scheduling, and application-signal autoscaling.

Warm-up0/3

1. A vLLM server starts with `--gpu-memory-utilization 0.9` and is serving one request. Roughly how much GPU memory does it report as used?Warm-up

A GPU memory bar with a dashed line marking the 0.9 config ceiling from --gpu-memory-utilization, and a box asking how much memory is reported as used while serving one request. — Setup: the flag's own ceiling, not the measured usage.

2. An 8-GPU inference node bills by the hour whether or not requests arrive. Which factor most directly sets your cost per generated token?Warm-up

An 8-GPU node box that bills by the hour whether idle or busy, next to a note that tokens generated this hour varies with traffic, and a box asking what sets cost per generated token. — Setup: the two quantities given in the stem.

3. A Kubernetes Pod is stuck in Pending because no node has enough free resources for it. What has to happen before it can run?Warm-up

Three nodes shown at 80%, 90% and 70% used capacity, a new Pod that requests more than any node has free and is marked Pending, and a box asking what has to happen before it runs. — Setup: a generic capacity shortage, no GPU specifics.

Check: The Problem0/1

1. A 70B model needs 8 GPUs across 2 nodes. Seven pods are Running, one is Pending, and the bill climbs while no request is served. Which missing ingredient allows this state?Analyze

Eight GPU pods for a 70B model, seven marked Running and one marked Pending, with text saying 7 are running and billing while no request is served, and a box asking what the missing ingredient is. — Setup: the partial-allocation state from The Problem, no candidate answers shown.

The Concept

Layer 1: node provisioning (Karpenter)

Karpenter watches pending pods and provisions nodes within ~45-60 seconds (Cluster Autoscaler typically takes 90-120 seconds for GPU nodes). It picks instance types dynamically per the NodePool constraint. If your pod needs 8 H100s and the cluster has no matching node, Karpenter provisions one directly instead of scaling an existing group.

The consolidation trap: Karpenter's default consolidationPolicy: WhenEmptyOrUnderutilized is dangerous for GPU pools. It can terminate a running GPU node to migrate pods to a cheaper right-sized instance. For inference that means evicting in-flight requests and reloading a 70B model on the new node, which costs minutes of capacity plus failed requests.

Safe setting for GPU pools:

yaml

disruption:
  consolidationPolicy: WhenEmpty
  consolidateAfter: 1h

This lets Karpenter consolidate truly empty nodes after an hour, while nodes running a job are left alone.

Layer 2: gang scheduling (KAI Scheduler)

KAI Scheduler (originally the NVIDIA Run:ai scheduler, now open source) handles what the default kube-scheduler does not:

Gang scheduling is all-or-nothing. A distributed inference job that needs 8 GPUs either starts all 8 together or none. Without it you get the partial-allocation trap: 7 of 8 pods start, then wait on the last one while the bill runs.

Topology awareness means knowing which GPUs share NVLink, which sit on the same rack, and which have InfiniBand between them, then placing pods to match. A DeepSeek-V3 671B tensor-parallel workload has to stay on one NVLink domain, and KAI Scheduler respects that.

Hierarchical queues let multiple teams share one GPU pool with priority and quota. Team B's training job can preempt Team A's production workload only if the priority rules allow it.

KAI runs alongside kube-scheduler as a secondary scheduler, and you annotate workloads to use it. Ray and vLLM production-stack both integrate with it.

Layer 3: application-level signals

The HPA trap: DCGM_FI_DEV_GPU_UTIL is a duty-cycle metric. It measures whether the GPU was doing work at each sampling interval, so 100% utilization could mean 10 concurrent requests or 100. The GPU was busy either way, which makes scaling on duty cycle scaling blind.

Memory doesn't help either. vLLM and similar engines pre-allocate KV cache memory (up to --gpu-memory-utilization), so usage stays near 90% even at one request. A memory-based HPA has no reason to scale down.

2026 replacement signals:

Queue depth (number of requests waiting for prefill).
KV cache utilization (what fraction of blocks are allocated to active sequences).
Per-replica P99 TTFT (your SLA signal).
Goodput (requests meeting all SLOs per second).

NVIDIA Dynamo Planner and llm-d Workload Variant Autoscaler consume these signals and scale replicas. For LLM serving they take the place of HPA.

When to use what

Scale decision	Tool
Add/remove nodes	Karpenter
Schedule multi-GPU jobs	KAI Scheduler
Add/remove replicas	Dynamo Planner / llm-d WVA (or custom HPA on queue depth)
Choose GPU type	Karpenter NodePool
Preempt low-priority	KAI Scheduler queues

Disaggregated prefill/decode complicates everything

If you run disaggregated prefill/decode (Phase 17 · 17), you have two pod classes with different scaling triggers: prefill pods scale on queue depth, decode pods on KV cache pressure. llm-d exposes these as separate Services with per-role HPA. A single HPA in front of both won't work.

Cold start matters here too

Cold-start mitigation (Phase 17 · 10) is where node provisioning time becomes user-visible. Karpenter's 45-60 second warm-up, plus a 20GB model load, plus engine init, puts a from-zero request at 2-5 minutes. Keep a warm pool (min_workers=1) for SLO-critical paths, or use Modal-style checkpointing at the application layer.

Numbers you should remember

Karpenter node provisioning: ~45-60s vs Cluster Autoscaler ~90-120s (GPU nodes).
KAI Scheduler prevents partial-allocation waste (the 7-of-8 trap).
DCGM_FI_DEV_GPU_UTIL as HPA signal: broken; use queue depth or KV utilization.
Karpenter WhenEmptyOrUnderutilized: terminates running GPU jobs. Use WhenEmpty + consolidateAfter: 1h for inference.

Check: The Concept0/24

1. What is Karpenter?Define

Setup: the 8-H100 example from The Concept, Layer 1.

2. How does Cluster Autoscaler relate to Karpenter?Define

Setup: Cluster Autoscaler's node-group model.

3. What is KAI Scheduler?Define

A kube-scheduler box and a Pod box that needs 8 GPUs across nodes, with a dashed box below asking what KAI Scheduler is. — Setup: the default scheduler and a multi-GPU pod.

4. What is gang scheduling?Define

Eight GPU pods for one job in a row, with no status shown for any of them, and a box asking what the placement rule is. — Setup: the 8-GPU job named in The Concept, Layer 2.

5. What does `DCGM_FI_DEV_GPU_UTIL` measure?Define

A GPU feeding a DCGM exporter, which reports a DCGM_FI_DEV_GPU_UTIL reading, and a dashed box asking what this reading measures. — Setup: where the metric comes from, not yet what it samples.

6. What is queue depth as an autoscaling signal for an LLM service?Define

An LLM replica serving requests next to a box named queue depth holding a count, and a dashed box asking what it counts. — Setup: the replica and the named signal, not yet what it counts.

7. What does KV cache utilization measure for a vLLM replica?Define

A vLLM replica serving requests next to a box named KV cache utilization holding a fraction, and a dashed box asking what it measures. — Setup: the replica and the named signal, not yet what it measures.

8. What does Karpenter consolidation do?Define

A mostly idle node with one small pod, a note that Karpenter is watching for savings, and a dashed box asking what consolidation is. — Setup: Karpenter observing a lightly used node.

9. What does `consolidationPolicy: WhenEmpty` with `consolidateAfter: 1h` do?Define

An empty node with 0 pods, a note that it stays empty for some duration, and a dashed box asking what consolidationPolicy WhenEmpty with consolidateAfter 1h does. — Setup: an empty node.

10. What does topology awareness mean for KAI Scheduler?Define

Four GPUs for one job, two on rack A and two on rack B, with a dashed NVLink line noted between them, and a box below asking what the placement rule is. — Illustrative rack layout for a 4-GPU job.

11. Which two tools consume queue-depth and KV-cache signals and take the place of HPA for LLM serving?Define

Setup: the two application-level signals named in The Concept, Layer 3.

12. Roughly how do Karpenter and Cluster Autoscaler compare on GPU node provisioning time?Explain

A pending GPU pod with two boxes, Karpenter and Cluster Autoscaler, each showing an unknown provisioning time, and a box asking how they compare. — Setup: the two autoscalers watching the same pending pod.

13. Why does a distributed inference job that needs 8 GPUs need gang scheduling in KAI Scheduler?Explain

Eight GPU pods for one job, seven marked Running and one marked Pending, with text saying 7 are running and billing while the 8th waits, and a box asking what role gang scheduling plays here. — Setup: the partial-allocation state from The Problem.

14. Why does a 100% `DCGM_FI_DEV_GPU_UTIL` reading fail to tell an HPA whether to add replicas?Explain

A box reading DCGM_FI_DEV_GPU_UTIL at 100%, next to an unknown backlog depth, and a dashed box asking why HPA cannot tell how deep the backlog is from this reading alone. — Setup: the stem's own 100% reading.

15. Why can a memory-based HPA not decide to scale a vLLM service down?Explain

Two boxes, one request in flight and busy with many requests, each showing an unknown memory reading, and a box asking why a memory-based HPA cannot scale this down. — Setup: the two load states compared in the stem.

16. Why is Karpenter's default `consolidationPolicy: WhenEmptyOrUnderutilized` risky for inference GPU pools?Explain

A running node actively serving requests, a note naming Karpenter's default consolidationPolicy WhenEmptyOrUnderutilized, and a dashed box asking what this policy can do to that node. — Setup: the stem's own policy name, applied to a running node.

17. A request arrives while a service is scaled to zero. Adding Karpenter's node warm-up, a 20GB model load, and engine init, how long until it is served?Explain

Three stages in a row: Karpenter node warm-up, a 20GB model load, engine init, with a box asking the total time until the request is served. — Setup: the three stages named in the stem, no durations shown.

18. Which Karpenter disruption setting keeps running jobs safe on an inference GPU pool while still reclaiming idle nodes?Apply

An empty node A and a running node B, with a dashed box asking for the safe disruption setting that reclaims idle nodes without touching running ones. — Setup: an empty node and a running node side by side.

19. Two teams share one GPU pool. Team B's training job should be able to preempt Team A's production service only when policy explicitly allows it. What provides this?Apply

Team A's production service and Team B's training job sharing one GPU pool, with a note that preemption is allowed only if policy explicitly says so, and a box asking what provides this. — Setup: the two teams and the policy condition from the stem.

20. An inference GPU pool uses Karpenter's default consolidation. Requests fail in bursts during quiet hours, each followed by minutes of reduced capacity, though nodes are healthy. What is the likely cause?Analyze

A quiet-overnight timeline with two failure bursts, each followed by minutes of reduced capacity, with a note that nodes report healthy, and a box asking the likely cause. — Setup: the overnight failure pattern from the stem.

21. Your vLLM Deployment has an HPA targeting `DCGM_FI_DEV_GPU_UTIL` at 70%. At peak, P99 TTFT reaches 12s, dashboards show 100% utilization, and the replica count never grows. What is the best fix?Analyze

An HPA targeting DCGM_FI_DEV_GPU_UTIL at 70%, with P99 TTFT at 12 seconds, the dashboard reading 100% utilization, and replica count flat, plus a box asking for the best fix. — Setup: the stuck-HPA scenario from the stem.

22. You run disaggregated prefill/decode pods (Phase 17 · 17). How should you autoscale them?Design

A prefill pods box and a decode pods box, each with an unknown scaling trigger, and a note that the two run as separate Kubernetes Services. — Setup: the two pod classes named in the stem.

23. You must run a 671B tensor-parallel model on 8 GPUs in a pool that a training team also uses. Which Layer 2 capabilities do you need?Design

A 671B tensor-parallel job needing 8 GPUs together, a training team sharing the same pool, and a box asking what Layer 2 capabilities are needed. — Setup: the shared-pool scenario from the stem.

24. A chat endpoint must meet its P99 TTFT SLO even for the first request after a quiet night. Karpenter warms a node in 45-60s. Which design fits?Design

A P99 TTFT SLO that must hold even for the first request of the night, a note that Karpenter warms a node in about 45 to 60 seconds, and a box asking what design fits. — Setup: the stem's own warm-up figure.

Use It

code/main.py simulates a three-layer autoscaler on a bursty GPU workload. It compares naive HPA (duty cycle), queue-depth HPA, and KAI-gang-scheduled scaling, and reports unmet requests, idle-GPU minutes, and mean wait.

Check: Use It0/4

1. In the simulator's QUEUE_DEPTH branch the replica target is 1 and 12 requests are waiting. The branch adds `max(1, qd // 5)`, capped at 16. What is the new target?Apply

The real QUEUE_DEPTH branch from main.py: if queue depth is over 5, target becomes target plus max of 1 and queue depth integer-divided by 5. Below it, target 1 and qd 12, asking for the new target. — Code copied from the lesson's main.py.

2. In the simulator's KAI_GANG branch the replica target is 1 and 12 requests are waiting. The branch adds `max(2, qd // 3)`, capped at 16. What is the new target?Apply

The real KAI_GANG branch from main.py: if queue depth is over 3, target becomes target plus max of 2 and queue depth integer-divided by 3. Below it, target 1 and qd 12, asking for the new target. — Code copied from the lesson's main.py.

3. In the simulator a scale-up is ordered at t = 600 s. Provisioning takes 50 s and the model load takes 45 s. About when can the new replica start serving?Apply

The real line from main.py, ready_at equals now plus NODE_PROVISION_SEC plus MODEL_LOAD_SEC, with the constants 50 and 45, and a scale-up ordered at t=600s, asking when it is ready. — Code and constants copied from the lesson's main.py.

4. On the author's run, the DUTY_CYCLE strategy drops fewer requests than QUEUE_DEPTH but wastes far more idle GPU minutes. What explains this?Analyze

Two bar pairs from a real run of main.py: dropped requests, DUTY_CYCLE 1 versus QUEUE_DEPTH 64, and idle GPU minutes, DUTY_CYCLE 266.2 versus QUEUE_DEPTH 49.5 minutes. A box asks what explains this. — Real output of main.py, fixed seed, both strategies on the same workload.

Ship It

This lesson produces outputs/skill-gpu-autoscaler-plan.md. Given cluster topology, workload shape, and SLO, it designs a three-layer autoscaling plan.

Author's Note

I'd run main.py before trusting the duty-cycle HPA claim in The Concept, because the output didn't match what I expected. The script is pure stdlib with a fixed seed, so the numbers are repeatable. On my run the DUTY_CYCLE strategy dropped fewer requests than QUEUE_DEPTH, at the price of far more idle GPU minutes. The cause is in how the simulator measures utilization: it counts busy replicas over ready replicas, which behaves more like a load signal than the sampled duty cycle the body warns about. Exercise 1 asks you to find where the difference comes from, and I think that mismatch is the answer to dig into.

The KAI_GANG branch is also thinner than its name. It reads the same queue depth as QUEUE_DEPTH, just with a lower threshold and a bigger step. Nothing in the code models gang scheduling or topology, so it shows aggressive scaling, not the 7-of-8 trap. The sim has no Karpenter consolidation either, and provisioning is a flat 50 seconds plus a 45 second model load per replica. The header says the latencies are illustrative, and I'd take that literally.

So my bet is that the simulator is useful for reading the scaling loops and weak as evidence for the three-layer argument. For the consolidation policy and gang scheduling, the Karpenter and KAI docs in Further Reading are the better test. If you extend the sim, I'd add a gang constraint first, since it is the layer the current code skips.

Exercises

Run code/main.py. Under a bursty workload, how many requests does naive duty-cycle HPA drop that queue-depth HPA catches? Where does the difference come from?
Design a Karpenter NodePool for a cluster serving Llama 3.3 70B FP8 on H100 SXM5. Specify capacity-type, disruption.consolidationPolicy, consolidateAfter, and a taint that keeps non-GPU workloads off these nodes.
Your team reports that deployments are stuck in Pending because "GPUs available but pod won't schedule." Diagnose it: is this Karpenter, kube-scheduler, or KAI Scheduler? Which metrics confirm?
Pick a signal to autoscale disaggregated prefill pods and a different signal for decode pods. Justify both.
Compute the cost of the WhenEmptyOrUnderutilized consolidation trap on a 24x7 production service that averages 60 request-dropping events/day at P99 TTFT > 10s.

Key Terms

Term	What people say	What it actually means
Karpenter	"the node provisioner"	Kubernetes node autoscaler; sub-minute provisioning
Cluster Autoscaler	"the old scaler"	Kubernetes node autoscaler predecessor; slower, group-based
KAI Scheduler	"the GPU scheduler"	Secondary scheduler for gang + topology + queues
Gang scheduling	"all or nothing"	Schedule N pods atomically or defer all of them
Topology awareness	"rack-aware"	Place pods based on NVLink/IB/rack placement
`DCGM_FI_DEV_GPU_UTIL`	"GPU utilization"	Duty-cycle metric; NOT a scaling signal for LLMs
Queue depth	"waiting requests"	Correct HPA signal for prefill-bound scaling
KV cache utilization	"memory pressure"	Correct HPA signal for decode-bound scaling
Consolidation	"Karpenter consolidation"	Node termination to cheaper instance type
`WhenEmpty + 1h`	"safe consolidation"	Policy that doesn't evict running GPU jobs