AI
The Node Is No Longer the Unit of Compute
Kevin Klues (NVIDIA) and Ellis Tarn (AWS) both named the same shift: Kubernetes was designed for small, fungible pods mapped to single nodes. AI inference has broken that model. A single inference request can require 8 hosts each carrying 8 GPUs, totaling 64 GPUs that must be co-located, low-latency interconnected, and scheduled together. Existing primitives for eviction, rollout, autoscaling, and lifecycle management were not built for workloads at that scale. Gang scheduling via tools like Kai Scheduler exists today, but the panelists want it integrated into core Kubernetes through SIG Scheduling.
How DRA Changed After Users Got Their Hands on It
The original “classic DRA” design handed all scheduling and allocation to an opaque, vendor-specific controller. Autoscalers had no visibility into what that controller was doing, so they could not predict which new nodes would satisfy pending resource requests. The current in-tree design makes device advertisement transparent to the scheduler. NVIDIA announced at the morning keynote that it is donating its DRA GPU driver to the CNCF, moving it to open governance so any customer or community member can file and merge fixes without NVIDIA holding veto power.
Cluster API as a Blueprint for Core vs. Extension
Zach Shepherd (Broadcom) pointed to Cluster API as the clearest working example of separating core from optional. It offers three layers: infrastructure providers that bind to a specific cloud vendor, ClusterClass objects that let teams encode repeatable configuration without copy-pasting it cluster by cluster, and runtime extensions for lifecycle hooks such as upgrading an add-on mid-cluster-upgrade or applying config changes without a full rollout. Dawn Chen framed the general principle as: mechanisms belong in Kubernetes core, policies belong in the wider project space where vendors can differentiate.
Incident Response: Upstream First, Cherry-Pick When Users Cannot Wait
Kevin Klues described NVIDIA’s rule: anything the team builds gets upstreamed, whether in NVIDIA-owned components or core Kubernetes. Zach Shepherd described the Broadcom process: retrospective to find true root cause, upstream the fix first, cherry-pick downstream when a customer cannot wait for the next release cycle. Repeated incident patterns prompt new shared tooling. Two recent examples are etcd-diagnosis and etcd-recovery, which help any team identify and repair a broken etcd cluster. The goal is reusable tools the whole community can pull from, not one-off patches held inside a single vendor’s fork.
What the Next Two Years Bring: Disaggregated Inference and Agentic Primitives
Dawn Chen predicted that within two years Kubernetes will schedule inference phases rather than inference requests, a shift that requires a new node abstraction layer distinguishing compute-dense nodes from memory-bandwidth-optimized nodes. NVIDIA is already investing in confidential computing, driven by agentic workloads that need hardware-level isolation. Ellis Tarn flagged agentic primitives, including sandbox support, as the area platform engineering teams will own. Dawn Chen filed a Kubernetes Enhancement Request the previous day proposing new primitives for AI workloads.
Q&A
Where would you start paying down Kubernetes technical debt? Kevin Klues said he would rewrite the Kubelet, which has grown organically over 12 years to the point where adding any new feature requires a full mental model of the entire codebase. ▶ 26:36
Beyond the Kubelet, what else would you redesign? Zach Shepherd identified API machinery, specifically the behavioral gap between built-in types and CRDs, as a source of code-path bifurcation that made less sense as the project matured. ▶ 27:28
Notable Quotes
the the thing that people are asking for the most is having the ability to gang schedule pods and make sure that all pods come up as a group rather than being scheduled individually Kevin Klues · ▶ 02:13
we’re talking about 64 GPUs all responsible for a single response Ellis Tarn · ▶ 06:50
mechanism cop mechanism is bound belong to the cubernetes core and uh uh but the policy policies belong to the ecosystem Dawn Chen · ▶ 10:56
I would rewrite the Kublet. Kevin Klues · ▶ 26:36
it’s kind of been organically growing over the course of what 12 years Kevin Klues · ▶ 27:00
Key Takeaways
- Gang scheduling 64 GPUs across 8 hosts for one inference request is unsolved in core Kubernetes today.
- DRA moved from opaque vendor controllers to transparent in-tree device advertisement after autoscaler failures surfaced in production.
- NVIDIA donated its DRA GPU driver to the CNCF so community members can merge fixes without vendor approval.
- Cluster API’s three-tier model (providers, ClusterClass, runtime extensions) is the panel’s reference design for extensibility.
- Upstream first, cherry-pick downstream: the practical incident response rule when users cannot wait for a release cycle.
About the Speaker(s)
Kevin Klues is a distinguished engineer on the NVIDIA Cloud Native team. He is a former shadow member of the CNCF Technical Oversight Committee and a sig-node maintainer, having led the design and implementation of DRA including the DRA driver for GPUs.
Dawn Chen is a principal software engineer at Google who worked on Kubernetes and Google Container Engine before the project launched publicly. She has served as a tech lead for both Kubernetes and GKE.
Rajas Kakodkar is a staff software engineer at Broadcom focused on low-level Kubernetes node functions. He is a tech lead of the CNCF Technical Advisory Group for Workload Foundation and co-chairs Cloud Native AI Day.
Zach Shepherd is a Distinguished Engineer at Broadcom and lead architect for the vSphere Kubernetes Service, where he sets the technical strategy for Kubernetes across the platform.