AI
The Inference Stack: From Single Node to Cluster Scale
Brian Stevens frames the open source AI challenge as three tracks: model capability, agentic patterns, and the serving stack. On the stack, vLLM launched 2.5 years ago and is now the most-used inference engine, accepted into the PyTorch Foundation. Single-node serving was the starting point, but production deployments run at cluster scale. That shift demands different scheduling algorithms. LMD and advanced gateway capabilities address exactly that layer, matching what frontier model operators already do internally.
Agentic Deployment: Security and Governance at Scale
Lin Sun points out that tools like Claude Code, Cursor, and Visual Studio Code already put AI agents on individual laptops. The hard problem is running those agents at scale with acceptable security and governance. Inside the cloud native community, a Kubernetes agentic networking working group is drafting a new API specifically for agentic traffic. Projects like Istio could implement that API to supply a security and governance layer. Human approval gates on critical steps are also part of what the emerging open source projects target.
Cross-Foundation Collaboration: Uber as a Real Production Example
Mark Collier argues against siloed foundation thinking. Uber, who spoke earlier in the day, trains and serves thousands of models using PyTorch, Ray, and vLLM together, then runs all of it on Kubernetes. Those projects live across the PyTorch Foundation and CNCF. LMD, a new CNCF project, is tightly integrated with vLLM from the PyTorch Foundation. Collier’s prescription: co-design and co-evolve projects across communities rather than ship disconnected pieces and expect users to assemble them.
Open Source Leading Proprietary Infrastructure
Stevens has a track record worth citing. He helped launch OpenStack 16 years ago, released Kubernetes into the CNCF, and brought LMD into the community. His read on the current moment: open source infrastructure and platform tooling is ahead of any single vendor or proprietary stack. He distinguishes this from open source model quality, where he expresses more uncertainty. On the platform side, he sees no apologies needed. The developers behind LMD also contribute to vLLM, K-gateway, and KServe, crossing project boundaries without walls.
Notable Quotes
the way the world could have gone is that every accelerator is building their own stack on how you actually operationalize AI inference or all the big model providers could be building the stack on how you actually operate operationalize AI inference. We took a different view and said that needs to be public domain Brian Stevens · ▶ 01:54
the biggest challenge right now is how we actually deploy and run AI agents at scale, at security, at the governance we feel comfortable. Lin Sun · ▶ 03:57
from an infrastructure and platform perspective, there’s no apologies for what’s being built here, right? It’s leading. It’s leading proprietary. It’s leading any single vendor. Brian Stevens · ▶ 09:28
anybody that tells you they know what’s going to happen a year from now, don’t believe them. Brian Stevens · ▶ 08:59
Key Takeaways
- vLLM, now 2.5 years old, is the de facto open source inference engine in the PyTorch Foundation.
- A Kubernetes agentic networking working group is drafting new APIs for agent security and governance.
- Uber runs thousands of models on PyTorch, Ray, vLLM, and Kubernetes simultaneously.
About the Speaker(s)
Brian Stevens is Red Hat’s SVP and CTO for AI, returning to Red Hat through the acquisition of Neural Magic, a company focused on inference. He previously helped release Kubernetes into the CNCF after leaving Red Hat for Google 11 years ago.
Lin Sun is Head of Open Source at Solo.io and served as KubeCon EU 2026 co-chair. She sits on the CNCF Technical Oversight Committee and maintains Istio, kgateway, and kagent.
Jonathan Bryce is Executive Director of Cloud and Infrastructure at the Linux Foundation, leading both CNCF and the OpenInfra Foundation.