AI
Why Attention State Makes Standard Load Balancing Wrong
In a decoder transformer, every token attends to all previous tokens. That state lives in the KV cache. Without it, cost grows as n-cubed. vLLM manages KV cache blocks like virtual memory, defaulting to block size 16 or 128, and uses automatic prefix caching to reuse blocks across requests by hashing token prefixes. As the KV cache hit rate rises, time-to-first-token drops. Prefill is compute-bound. Decode is bandwidth-bound. Those two regimes need different handling, which is the foundation for everything llm-d does.
Round-Robin Routing Destroys Distributed KV-Cache
In production you run multiple vLLM replicas. The single-instance KV cache is now split across pods. With round-robin scheduling, a multi-turn chat request hits pod A on turn one, then pod B on turn two. Pod B recomputes the entire context from scratch. GPU memory holds duplicate state. TTFT rises. On a 16 H100 benchmark where the workload demands only 73% of distributed KV-cache capacity, naive load-aware scheduling causes cache thrashing and waiting queues that consume all headroom. The system cannot serve, even though it should have capacity to spare.
llm-d Inference Scheduler: Prefix-Cache-Aware Routing at Scale
llm-d’s inference scheduler balances two routing objectives. Prefix-cache-aware routing tracks which pods hold which cached prefixes and sends requests there, creating stickier affinity. Load-aware routing watches queue depth, running request count, and KV-cache utilization to avoid hotspots. The endpoint picker scrapes metrics from vLLM pods every 200 milliseconds (configurable) and runs as an Envoy external process via the Kubernetes Gateway API Inference Extension. On the same 16 H100 benchmark, prefix-cache-aware routing reaches the theoretical throughput limit while keeping waiting queues near zero.
Prefill/Decode Disaggregation and KV-Cache Offloading
llm-d ships two additional well-lit paths. Prefill/decode disaggregation splits the two compute phases into separate vLLM deployments. A typical configuration runs several small prefill instances feeding one large decode instance, with the endpoint picker dynamically choosing whether a given request uses disaggregated PD. At 96 GPUs on H200 hardware, the llm-d 0.3 blog showed 2.2K tokens per second per GPU. KV-cache offloading extends the GPU cache pool northward into CPU memory and storage. In a saturation benchmark, storage offloading sustains throughput across growing request counts without adding hardware.
DaoCloud Token Factory: On-Premise Deployment with DeepSeek
DaoCloud builds on-premise token factories, hardware-plus-software stacks tuned to maximize token output. Their production architecture for DeepSeek uses tensor parallelism 16 with 8 GPUs per node, two prefill instances and one decode instance, with NVLink exchanging KV-cache data between prefill and decode. The four-layer stack is: llm-d inference orchestration for gateway scheduling and autoscaling; vLLM as the inference engine; flash attention and interconnect kernels; and Kubernetes for hardware adaptation including Nvidia GPU Operator. DaoCloud benchmarks with real traffic patterns and feeds results back into the next tuning cycle.
Hands-On Tutorial: Kubernetes Service vs. llm-d Scheduler
The tutorial runs entirely on a local kind cluster with no real GPUs. The llm-d inference simulator replaces vLLM, simulating KV-cache behavior including prefix cache hits that reduce latency. Part one deploys eight simulated Mistral pods behind a standard Kubernetes service and runs the llm-d benchmark with a 6,000-token system prompt, 150 users each sending 1,000-token questions with 1,000-token outputs. Part two replaces the service with the llm-d inference scheduler and reruns the same benchmark. The naive deployment caps around 40,000 total tokens processed. The llm-d deployment roughly doubles that figure.
Q&A
How does the Gateway API Inference Extension fit into the architecture? The extension adds an inference pool as a backend; Envoy forwards requests to the endpoint picker (the llm-d scheduler) via the external process protocol, which selects a pod and returns it to Envoy for direct pod communication. â–¶ 60:02
Can an intelligent semantic router for model selection sit above llm-d? Yes, projects like the vLLM semantic router classify requests and route to different model pools; llm-d handles per-instance optimization within a pool, and a new llm-d sub-project will show those integrations. â–¶ 1:16:37
Does llm-d handle heterogeneous short and long context workloads? llm-d routes based on queue length, KV-cache fullness, and connected request counts; an SLO-aware router using an XGBoost model trained at inference time is in development for finer-grained placement. â–¶ 1:20:03
Notable Quotes
you’ll be transforming your cash bills into cash hits Maroon Ayoub · ▶ 04:55
if you don’t have KB cache every token you have you have to recomputee everything uh previously so it would really be n cubed if you didn’t have KV cache Tyler Michael Smith · ▶ 08:02
naive or only load aware scheduling destroys your throughput and you know you’ll get cash thrashing Maroon Ayoub · ▶ 22:54
LLMD’s job is optimizing instances of VLM in a cluster Tyler Michael Smith · ▶ 1:16:42
useful or or or common production workloads are heterogeneous in in nature. So it’s not uh the same input or output ratios or lengths and you have multiple VM instances. So that’s where LMD comes in. Maroon Ayoub · ▶ 1:19:24
Key Takeaways
- Naive round-robin routing on 16 H100s causes cache thrashing even when workload fits in 73% of capacity.
- llm-d’s prefix-cache-aware scheduler reaches theoretical throughput ceiling on that same benchmark.
- Prefill/decode disaggregation on H200s hits 2.2K tokens per second per GPU at 96 GPUs.
- KV-cache offloading to CPU and storage sustains throughput under growing load without adding hardware.
- The full llm-d stack runs locally on kind with no GPUs using the llm-d inference simulator.
About the Speaker(s)
Maroon Ayoub is a systems engineer at IBM Research focused on distributed AI infrastructure. He co-leads development of llm-d and specializes in scaling LLM inference with Kubernetes-native architectures, performance efficiency, and open source integrations.
Tyler Michael Smith holds a PhD in Computer Science from The University of Texas at Austin, where he studied high-performance dense linear algebra, microkernels, parallelism, and theoretical lower bounds on data movement. After a postdoc at ETH Zürich he joined Neural Magic and now serves as Chief Architect for Inference Engineering at Red Hat. He is a vLLM core maintainer and co-leads the llm-d PD disaggregation SIG.
Kay Yan is a Principal Software Engineer at DaoCloud with 15+ years of hands-on experience in Kubernetes and AI/LLM workload orchestration. She maintains four CNCF Sandbox projects, including Kubean, Spiderpool, HwameiStor, and Merbridge, and leads DaoCloud’s token factory work built on llm-d.
Vita Bortnikov is a Distinguished Engineer at IBM whose work centers on distributed systems with emphasis on high availability and scalability. She leads IBM’s Microservices Framework team and is among the Istio project’s core contributors, with additional focus on Kubernetes-based VM workload management.
Nili Guy is a Research Manager and Senior Technical Staff Member at IBM Research. She is a co-creator of llm-d and has led key open-source and productized inference initiatives across IBM’s AI platforms, with deep expertise in distributed inference and Kubernetes-native AI systems.