AI
Why Standard Load Balancing Fails AI Workloads
Traditional web apps are lightweight and predictable. AI models are memory-hungry and serve non-uniform workloads. That mismatch breaks conventional traffic routing. The result is delayed responses, hardware sitting idle, and cloud bills rising fast. Sixty percent of organizations already run Kubernetes for AI, according to a stat cited the previous day at KubeCon. The existing plumbing was never designed for this load profile, and patching it does not fix the fundamental mismatch.
Sovereign AI Means Interoperability, Not Isolation
79% of organizations rank sovereign AI as a top strategic priority. Red Hat’s position: sovereignty means avoiding vendor lock-in, not cutting off external systems. The answer is a Kubernetes-native AI factory built on an open-source foundation. That foundation gives operators technical and operational control over the full stack. The modular design lets teams deploy AI in a public cloud, at the edge, or inside a sovereign cloud. It also supports regulatory mandates like the EU AI Act.
The Three-Layer Inference Stack: KServe, Gateway API, and LLMD
Red Hat built its inference stack on vLLM and LLMD, integrating both natively into Kubernetes. KServe sits at the top as an abstraction layer, handling production-ready autoscaling and hiding underlying complexity. The Gateway API Inference Extension adds AI-aware load balancing on top of standard Kubernetes networking. LLMD sits below both, acting as the orchestrator for distributed resources. Together the three layers convert a standard cluster into what the speakers call a sovereign inference engine.
Prefix-Aware Routing and the KV Cache
LLMD upgrades Kubernetes with an AI-aware routing fabric. Instead of basic HTTP routing, the inference gateway inspects the KV cache. Prefix-aware routing directs each prompt to the node that already holds that context in memory. That cuts time-to-first-token by removing the need to re-run GPU processing for cached prefixes. LLMD also handles prefill and decode disaggregation, separating the compute-heavy prompt-reading phase from the memory-bound token-generation phase across different nodes in the cluster.
Multi-Dimensional Scheduling Beyond Round Robin
Simple round-robin scaling treats every request as identical. AI inference is not identical request to request. LLMD applies multi-dimensional, state-aware scheduling that tracks GPU memory occupancy, cache state, and workload phase simultaneously. The goal is squeezing performance from expensive GPUs that sit underutilized under naive schedulers. Prefill and decode disaggregation runs across the cluster dynamically, so neither phase blocks the other. The combination of prefix routing and phase disaggregation is what separates this stack from a basic Kubernetes deployment.
Notable Quotes
We have reached a tipping point. Karena Angell · ▶ 00:09
true sovereignty doesn’t mean isolation. Vincent Caldeira · ▶ 01:57
We’re moving far beyond simple round robin scaling to multi-dimensional state-aware scheduling that squeezes every ounce of performance out of your expensive GPUs. Vincent Caldeira · ▶ 05:06
Key Takeaways
- 79% of organizations treat sovereign AI as a top strategic priority, driving demand for open, portable inference infrastructure.
- KServe, the Gateway API Inference Extension, and LLMD together form a three-layer Kubernetes-native inference stack built on vLLM.
- Prefix-aware KV-cache routing and prefill-decode disaggregation cut time-to-first-token and GPU waste simultaneously.
About the Speaker(s)
Vincent Caldeira is Red Hat’s APAC CTO and an Industry Visiting Scholar at Columbia University. Named a Top 10 APAC CTO in 2023, he brings more than 20 years of finance IT experience and focuses on open source, cloud-native technologies, and AI strategy across the Asia-Pacific region.