Why LLM Workloads Break Standard Kubernetes Assumptions

▶ Watch (0:03)

LLMs are not web applications. Their containers are large, their connections are long, and their hardware requirements are specific. Sghiouar frames the talk around teams that lack unlimited GPU access. Running DeepSeek at 671 billion parameters means handling a 1.37 terabyte model at full 16-bit precision. That physical scale forces operators to think across multiple GPUs and sometimes multiple hosts before they write a single line of deployment config.

Three Optimization Layers: Model, Serving Engine, Infrastructure

▶ Watch (8:10)

Sghiouar organizes the problem into three stacked layers. First, the model itself: quantization sets precision and controls memory footprint directly. Lower precision means less memory. Second, the serving engine: vLLM, Hugging Face TGI, Nvidia NIM, Triton, Ray Serve, and Ollama each have different compatibility matrices against model types and accelerator families. Most support CUDA but not TPUs. Third, the infrastructure layer targets the accelerators themselves. Horizontal concerns like pre-warming, caching, and autoscaling sit across all three.

Storage and Cold-Start: The Container Size Problem

▶ Watch (11:56)

The vLLM OpenAI container, with no model inside it, is 5.24 GB and takes roughly five minutes to pull on GKE. Add a 1.37 terabyte DeepSeek model and the 5 GB becomes irrelevant, but the download time becomes an autoscaling blocker. Spegel, an open source project, turns every Kubernetes node into a mirror so adjacent nodes pull cached images locally rather than from a remote registry. Pairing Spegel with OCI container images mounted as volumes, available in beta since Kubernetes 1.35, reduces repeated downloads further.

Dynamic Resource Allocation: GPU Claims Beyond “I Want 3 GPUs”

▶ Watch (17:01)

The existing device plugin lets a pod request a count of GPUs but not a specific model or memory size. Dynamic Resource Allocation (DRA) mirrors the storage CSI pattern: an operator publishes a resource slice describing an Nvidia GPU with 40 GB of memory and a specific model ID, then a workload files a resource claim requesting any GPU with at least 30 GB. The scheduler matches claim to slice. DRA also supports network interface claims. Nvidia open-sourced their GPU drivers this week; Google is open-sourcing TPU drivers to pair with the same API.

Intelligent Routing with the Gateway API Inference Extension

▶ Watch (22:56)

Round-robin load balancing fails for LLMs because requests are non-uniform. A multimodal request carrying a video costs far more than a short text prompt. Traditional load balancers route on path, not on request body. The Gateway API inference extension introduces two new objects, InferencePool and InferenceObjective, plus an endpoint picker extension. The picker reads the model ID from the request body and returns the correct backend pool to the gateway. The LLMD project, donated to the CNCF this week, extends this with KV cache utilization and prefill/decode pool splitting.

Q&A

Does the endpoint picker extension add meaningful latency? The picker operates in microseconds. ▶ 30:02

For 30 GB images, does Bottlerocket help cold-start times? Sghiouar had not tested Bottlerocket directly, but recommended image streaming on EKS, noting the second pool pull is usually faster than the first. ▶ 30:26

How do you handle bin-packing fragmentation across a large heterogeneous cluster? Use a single network-attached volume mounted to every node rather than pre-loading images per node, then rely on standard Kubernetes node selector and toleration mechanisms to land jobs on the right hardware. ▶ 31:38

Notable Quotes

now you have to run Olama with like Gemma 3 and the model the container is like 6 GB, right? So, good luck autoscaling that very quickly. Abdel Sghiouar · ▶ 08:58

if we look at the deepseek which is all the way at the bottom at full quantization which is 16 bits it’s roughly 1.37 terabytes Abdel Sghiouar · ▶ 13:26

optimizing large language models on Kubernetes is a multi-layer problem. It’s a multi-dimensional problem. You shouldn’t think about it as optimization in a single place. Abdel Sghiouar · ▶ 29:08

Key Takeaways

  • DeepSeek at full 16-bit precision is 1.37 terabytes; autoscaling requires solving storage first.
  • Spegel and OCI volumes as Kubernetes mounts reduce repeated large image downloads across nodes.
  • DRA replaces blunt GPU counts with typed resource claims matching specific memory and hardware models.
  • The Gateway API inference extension routes by request body, not path, fixing uneven backend load.
  • LLMD, now a CNCF project, packages inference gateway, body-based routing, and KV-aware scheduling together.