AI
What Swap Gets Wrong: The Benchmark Numbers
A 5B model running CPU-only inference with unlimited swap saw P50 latency jump from 40 seconds to 116 seconds per request. That is 8x degradation. The kernel swapped 1.6 GB to disk, generating roughly 100,000 page-ins and 400 page-outs. Every inference step waited on disk I/O instead of computing. On GPU, Mistral 7B under the same swap pressure dropped GPU utilization from 94% to 73.5%, a 21-percentage-point collapse, while latency rose 26%. The GPU was waiting on the CPU, and the CPU was waiting on disk.
Limited Swap in Kubernetes 1.34: What Changed
Unlimited swap, available through Kubernetes 1.29, let any container consume any amount of swap with no ceiling. One misbehaving best-effort pod exhausted a 60 GB swap device in under three minutes in Vermande’s tests, taking the entire node down. Kubernetes 1.30 removed unlimited swap entirely. Limited swap, now stable in 1.34 as of August 2025, is cgroup v2-aware and container-scoped. Burstable pods get swap proportional to their memory request divided by node capacity times total node swap. Guaranteed pods, where requests equal limits, get zero swap. Best-effort pods also get zero, because no request means no allocation under the formula.
The Three-Layer Memory Hierarchy for AI Inference
AI inference stacks operate across three memory tiers. GPU VRAM, connected via HBM at 2 to 3 TB/s, holds model weights and active KV cache entries. The vLLM CPU memory pool acts as an L2 cache. When VRAM fills, the page-attention mechanism moves less-recently-used KV cache blocks over PCIe at roughly 64 GB/s into CPU RAM. When CPU RAM exhausts, the kernel pages to disk. NVME gives 3 to 5 GB/s. Spinning drives give 100 to 200 MB/s, a 1,000x difference. Application-level eviction handles the common case. Kubernetes swap handles the failure case.
Scenario One and Two: When Swap Saves You Money and Your Uptime
Running three inference services (text classification, translation, summarization) on a single node with 35 GB of combined base memory against a 40 GB node limit, simultaneous spikes triggered OOM kills lasting two to three minutes. Adding a second A100 node on GKE costs roughly $2,700 per month purely for headroom. Burstable QoS with NVME-backed limited swap eliminated all OOM kills across the test window. In a second scenario, a 3x traffic spike exhausted both VRAM and the vLLM CPU swap pool. Without Kubernetes swap as an L3 safety net, the chatbot went down at peak demand. Correctly sized swap kept it alive, with latency increases within acceptable SLA bounds.
Scenario Three: When Swap Makes Things Worse
Swap does not help every workload. Sub-millisecond SLA workloads like speech recognition and real-time translation cannot absorb the added latency. Training workloads with sequential memory access patterns are also poor candidates. A LoRA fine-tuner under constrained memory showed only a 1.01x overhead at small scale, but Vermande notes large-scale training degrades significantly more. The worst outcome is undersized swap: 60% latency overhead followed by an OOM kill anyway. That is strictly worse than letting the OOM kill happen immediately. For latency-sensitive inference, use guaranteed QoS, which sets memory swap max to zero at the cgroup level.
Decision Framework and Monitoring Thresholds
Three metrics matter once swap is enabled. Container memory swap, where above 50% for five minutes signals trouble. Node memory swap free, where an alert below 20% remaining capacity gives time to react. Container swap I/O count for detecting thrashing. On a KubeDM cluster, Vermande measured 200,000 page-ins for vLLM, 82,000 for Whisper, and 372,000 page-outs during training. The kubelet configuration is four lines. Verify per-container cgroup v2 settings after applying. Always use NVME, never spinning disk. And always set memory requests explicitly, because removing them zeros the swap allocation entirely under the proportional formula.
Notable Quotes
Eight times latency degradation. Every inference request can be eight times slower. 1.6 GB of memory threshing between RAM um and disk on a single misconfigured Kuber Kubernetes nodes. Nic Vermande · ▶ 00:41
should I use swap it should be which failure mode do I prefer? degraded but alive or completely dead right that’s essentially that Nic Vermande · ▶ 07:33
one misbehaving best effort pod could exhaust swap in under three minutes on a 60 gig swap device. So one misconfiguration a whole note down. Nic Vermande · ▶ 08:48
swap in kubernetes is no longer a question of should we uh it’s a question or where how much and for which workloads you now have the data to answer that Nic Vermande · ▶ 34:14
Key Takeaways
- Unlimited swap is gone in Kubernetes 1.30+; limited swap in 1.34 uses cgroup v2 per-container caps.
- Correctly sized NVME-backed limited swap eliminates OOM kills during multi-model and traffic-spike scenarios.
- Undersized swap produces 60% latency overhead before an OOM kill, worse than disabling swap entirely.
About the Speaker(s)
Nic Vermande is a Developer Advocate at ScaleOps with 21 years of hands-on experience across Cloud-Native technologies, Open Source Software, Virtualization, and Datacenter networking. He focuses on enabling users and building technology that solves real operational problems, and speaks regularly at global tech conferences and online events.