AI
Why Inference Routing Needs Body Inspection
Standard Kubernetes services use L4 load balancing. They know nothing about prompt content or LoRA adapter names. Inference Gateway adds an endpoint picker service that sits alongside the L7 proxy (Envoy) and reads the request body. It uses the xRPC extension protocol to pass routing hints back to Envoy, which then forwards to the model server the picker selects. This gives the system a global view of model server state, something an L4 service cannot provide.
Prefix-Aware Routing and What It Buys
Two prompts sharing a common prefix, “What is the weather like in Amsterdam?” and “What’s the weather like in Phoenix?”, can reuse the same KV cache entries if routed to the same model server. Inference Gateway tracks prefix matches across the pool and routes accordingly, cutting prefill time and freeing HBM for decode. Benchmarks with a heavy prefill workload show a clear delta in output tokens per second and a lower time-to-first-token compared to a plain Kubernetes service baseline.
The Thundering Herd Problem with Static Weights
Prefix scoring magnetizes traffic toward whichever server holds a hot cache entry. With a long shared system prompt, one model server gets hammered while others sit idle. The current framework uses manually tuned weights to balance affinity scores against distributive scores (KV cache utilization, queue depth, running requests). If those weights are off, the hot node problem appears. Asking end users to tune these weights is the wrong answer. The framework was built for contributors, not for operators configuring production traffic.
Adaptive Routing: Two New Components
Nir Rozenbaum’s design adds two components to the control plane, leaving the data plane request path untouched. The async detector periodically reads per-endpoint metrics, KV cache utilization, waiting requests, and active requests, then emits a signal between zero and one using normalized coefficients of variation. The adaptive configurator consumes that signal, applies a generalized sigmoid function for smooth monotonic translation, and adjusts scorer weights. Asymmetric thresholds prevent flip-flop when entering and exiting states like burst mode.
Benchmark Results: Herd Problem Resolved
Near ran a low-concurrency baseline with very high prefix match to simulate a hot node, then spiked to higher concurrencies. At concurrency 125, the control group (default Inference Gateway config) performs significantly worse because the hot node is overloaded. Adaptive routing holds. At very high concurrency, both approaches converge, but adaptive handles more traffic patterns without any configuration change from the user. A separate baseline run against published LMDuler benchmarks shows no regression: time-to-first-token and inter-token latency stay within error margins. The control group also dropped more requests across three runs.
What Comes Next
Three directions are active. First, more benchmarking against production workloads, not just showcase benchmarks. Second, a predictive latency approach from teammates at LMDuler uses a small ML model to predict request latency and stack-rank endpoints. A blog post dropped the day of this talk. Third, flow control with saturation detection runs before scheduling and could tell the system when individual endpoints are full, which may make the weighting problem easier to solve. Prefill-decode disaggregation is also being evaluated as another way to decompose the routing challenge.
Notable Quotes
inference routing requires looking at the body specifically for things like prompt content or the model name if you’re doing uh PEFT routing like Laura. Kellen Swain · ▶ 01:02
we built a pluggable framework that allows us to make decisions, create routing algorithms, and uh be wrong so other people can come up with other ideas. Kellen Swain · ▶ 02:36
We want this to reliably work out of the box and adding new novel scores should someone decide to contribute should be easy and the weight management should be easy as well. Kellen Swain · ▶ 07:01
want adaptive routing to be deterministic and explainable. So if we look at the dashboard and see behavioral changes, we want to know why. Nir Rozenbaum · ▶ 09:02
the core first commit was vibe coded and told to look at inference gateway. It does work. Don’t worry, it’s not it’s not horrible code, but it was generated. Kellen Swain · ▶ 19:27
Key Takeaways
- Prefix-aware routing cuts prefill time by sending matching prompts to servers that already hold the KV cache.
- Static scorer weights cause thundering herd failures at medium concurrency; adaptive weighting fixes this without user configuration.
- The async detector and adaptive configurator run entirely in the control plane, adding zero per-request overhead.
About the Speaker(s)
Kellen Swain is a software engineer at Google on the GKE inference team and a maintainer of the Kubernetes Inference Gateway project.
Nir Rozenbaum is a Senior Principal Software Engineer at Red Hat, a maintainer of Kubernetes Inference Gateway, and chair of the Kubernetes AI Gateway Working Group. He focuses on AI/ML workloads on Kubernetes and contributes to CNCF standards for cloud-native inference.