AI
KServe: One Platform for Predictive and Generative AI
KServe runs on Kubernetes and handles both predictive AI and generative AI inference from a single platform. It supports serving runtimes including vLLM for large language models and XGBoost and scikit-learn for predictive workloads. It integrates with LMD, Envoy AI Gateway, and KV Cache for generative AI use cases. Gateway API support makes it compatible with Istio and Envoy. Because it runs on Kubernetes, it deploys wherever Kubernetes runs.
CNCF Incubation and Community Growth
KServe joined the CNCF as an incubating project in September 2024. The project now has 19 maintainers and over 300 contributors accumulated across its history. Organizations are actively adopting and contributing to the project. That growth reflects demand from teams running model inference in production across both predictive and generative workloads.
Predictive AI Features Still Supported
KServe began as a predictive AI serving project and those capabilities remain. Scale-to-zero and request batching reduce idle compute costs. Security integrates through OPA and OAuthz. Distributed tracing, autoscaling based on GPU and CPU usage, request and response logging, traffic management, and out-of-box metrics are all available. Teams with existing predictive workloads do not need to migrate off the platform to add generative AI inference.
Generative AI Scaling and Efficiency
For large language models, KServe adds metrics-based autoscaling through KEDA. Autoscaling pulls metrics directly from serving runtimes. Supported generative AI runtimes include vLLM, TRT-LLM, and LMD, each improving model serving efficiency. Prefix caching reduces cost and improves throughput. Intelligent routing and traffic management work alongside caching to handle bursty LLM traffic. The llm-d project, recently accepted as a CNCF sandbox project, provides additional model inference optimizations for large language models.
Getting Involved with KServe
KServe hosts bi-weekly community meetings on Thursdays. Two dedicated Slack channels serve end users and contributors separately. At KubeCon EU, a maintainer session runs on Thursday and a project booth is open Tuesday, Wednesday, and Thursday mornings. The repo and project website links are available on the talk slides. Contributors with technical questions can reach the team through the contributors Slack channel.
Notable Quotes
Kserve is a standardized, distributed and scalable model inference platform on Kubernetes for both predictive AI and generative AI. Yuan Tang · ▶ 00:37
we joined the cncf as an incubating project uh in September last year and it’s amazing to see so planning uh community adopters and organizations adopting and contributing to Kerve. Yuan Tang · ▶ 01:28
we also support pump caching to reduce the cost and also improve the throughput through features like intelligent routing and traffic management. Yuan Tang · ▶ 03:16
Key Takeaways
- KServe joined CNCF as an incubating project in September 2024 with 19 maintainers and 300-plus contributors.
- Prefix caching and metrics-based KEDA autoscaling reduce cost and handle LLM traffic spikes.
- Predictive AI features like scale-to-zero and request batching remain fully supported alongside generative workloads.
About the Speaker(s)
Yuan Tang is a Senior Principal Software Engineer at Red Hat AI and project lead for KServe. He holds leadership positions across several open source communities including Argo, Kubeflow, KServe, Kubernetes, and CNCF. He is a maintainer and author of multiple popular open source projects and has authored three technical books.