AI
KServe as a Kubernetes Inference Control Plane
KServe sits above Kubernetes and handles everything needed to run models at production grade. It abstracts networking, security, ingress, and egress so data scientists do not have to configure Kubernetes directly. For autoscaling the project moved from Knative to KEDA. Envoy AI Gateway handles networking. vLLM handles inference speed. KServe started with predictive workloads, covering classifiers and regression models using scikit-learn, XGBoost, and LightGBM, and has since expanded to generative AI with a dedicated controller and CRD added roughly six months before this talk.
CNCF Incubation and the LMD Integration
KServe joined the Linux Foundation and LF AI Data Foundation, then was accepted as a CNCF incubating project. CNCF acceptance signals governance, security, and trustworthiness to the wider community. On the inference side, LMD joined CNCF sandbox within one year of release and was already at version 0.6. KServe tracks LMD releases closely, shipping each new version into the platform as it lands. PR volume into the KServe repo has grown roughly ten times compared to earlier periods, reflecting the surge in contributor interest.
KV Cache, KEDA, and Autoscaling for LLMs
GPUs do not handle concurrency natively. A request arrives, the GPU processes it, and returns. vLLM closes that gap by sharing a KV cache across nodes, so repeated or similar prompts skip recomputation entirely. KEDA replaces basic HPA for autoscaling because it accepts any Prometheus query, allowing KV cache utilization to drive scaling decisions. When cache fills fast, depending on prompt size and GPU memory, KEDA can react before throughput collapses. For deeper control, WVA (Workload Variant Autoscaler) adds pluggable actuators and centralizes all scaling logic in one CRD.
Prefill-Decode Disaggregation with LMD
LMD splits prefill and decode into separate pods because the two phases have different compute profiles. Prefill builds the KV cache and is compute-heavy. Decode generates tokens using that cache and is memory-bandwidth-bound. Separating them prevents each phase from starving the other. Pods communicate over NVLink when on the same node or over high-speed RDMA networking across nodes. The NIXL library, developed by Nvidia, abstracts both transports. The inference scheduler routes each request and decides whether to disaggregate based on current cluster state.
Demo: LMD Deployment via the LLMInferenceService CRD
The demo showed a 20-billion-parameter GPT OSS model deployed with prefill-decode disaggregation on GPU MIG partitions. Two prefill vLLM pods and one decode pod ran alongside a scheduler pod hosting the endpoint picker. KServe’s operator created all supporting objects, including RBAC, config maps, and the Envoy gateway configuration, from a single LLMInferenceService CRD. Model weights were downloaded once to a persistent volume to avoid redundant pulls per pod. Prometheus and Grafana dashboards exposed endpoint picker metrics, vLLM metrics, and Nvidia DCGM GPU metrics split by prefill and decode workload.
Notable Quotes
LLMD is a Kubernetes native higherformance distributed LLM inference framework. It orchestrates VLLLM in a variety of ways to bring state-of-the-art inferencing performance through several optimizations. Filippe Spolti · ▶ 25:53
GPU is not cheap. is uh it’s really difficult to get it even though in the company wise it’s when I request it normally it take one day two today to take it if I take it I don’t release it Jooho Lee · ▶ 10:38
send it the PR like a 10 times more than before. Jooho Lee · ▶ 31:56
Key Takeaways
- KServe v2 adds a dedicated LLMInferenceService CRD covering generative AI workloads alongside predictive ones.
- Prefill-decode disaggregation on LMD reduces inter-token latency by separating compute and memory phases into dedicated pods.
- KEDA replaces Knative autoscaling, enabling KV cache utilization as a first-class autoscaling metric for GPU efficiency.
About the Speaker(s)
Filippe Spolti is a Senior Software Engineer at Red Hat leading work within the OpenShift AI Group. His background spans Business Automation and Cloud solutions, and he now focuses on AI infrastructure on Kubernetes using KServe and ModelMesh.