AI
DRA at GA: What the Core API Delivers
Dynamic Resource Allocation replaces the device plugin interface with two paired APIs. The ResourceSlice API lets DRA drivers publish device metadata to the scheduler. Users write ResourceClaims to say what they need, such as a GPU with a specific memory size or alignment with another device. The scheduler picks a matching device, the kubelet receives the allocation, and the container runtime activates it via CDI or NRI. DRA Net, for example, skips CDI entirely and works through NRI.
Active Work in 1.36: Partitionable Devices, Preemption, and Scheduling
Partitionable devices reached beta in 1.36 after an API redesign. The 1.34 structure forced all partitions sharing an underlying resource into a single object, hitting etcd’s size limits. The fix decouples partitions so they scale arbitrarily. A KEP for preemption of DRA-allocated pods is targeted at 1.37; right now those pods cannot be preempted. SIG Scheduling is also building pod-group and gang-scheduling support for multi-node GPU and TPU training jobs, and WG Device Management is integrating with that effort.
Vendor Driver Donations: Nvidia GPU and Google TPU
Nvidia donated its GPU DRA driver and compute-domain DRA driver to the Kubernetes project. Google donated a TPU driver, announced on the Google Cloud blog. A CPU DRA driver also now exists, enabling topology-aware alignment of GPU, NIC, and CPU within the same NUMA node. The Nvidia driver is not yet GA, but a future release will let ResourceClaim parameters dynamically configure multi-instance GPU partitions, time slicing, and MPS sharing at pod scheduling time.
Expanding the Resource Model: Native Resources and Workload-Aware Scheduling
CPU and memory are not yet under DRA management. They are still declared in the PodSpec, where the topology manager resolves alignment at kubelet time. That late binding can place a pod on a node where the policy cannot be satisfied, causing a failure. Moving those decisions to the scheduler is the goal. Native resources in DRA are initially targeted at workloads like Slurm that need GPU, NIC, and CPU aligned on the same PCIe bus or NUMA node, not broad deployment across all nodes.
Growing the Contributor Base
Several KEPs merged for 1.36 came from first-time DRA contributors. Noral Dean implemented the resource pool status API, which lets users query device availability and current allocations. Allay added metadata injection into containers so tools like kubevirt can discover which devices are present. Pravin designed the native resources KEP alongside the CPU DRA driver. A KEP is a heavyweight process requiring design defense. Patrick Ohly noted that building this wider contributor base is a priority, and unowned enhancement issues are open for new participants.
Q&A
Does DRA replace node labels from Node Feature Discovery? DRA can already eliminate node labels used only for scheduling GPU types, since ResourceClaims carry that metadata directly. Broad replacement of NFD labels via native resources in DRA is possible eventually but still early. ▶ 25:10
How does the working group handle device failure at day two? Patrick implemented taints and tolerations for devices so a driver can taint a degraded device, evicting pods using it or blocking new allocations. Propagating a failure signal up through the resource slice to the consuming application is an idea without a KEP yet. ▶ 28:46
Why switch from extended resources to ResourceClaims? Extended resources remain fine for simple count-based GPU scheduling. ResourceClaims add prioritized selection, cross-device constraints, consumable capacity, and flexible sharing between pods and containers. Features like MIG and time slicing require that expressive power. ▶ 30:26
Notable Quotes
Hardware is super complex. We don’t need to model the hardware exactly. We need to model kind of the scheduling constraints on that hardware. John Belamaric · ▶ 08:23
probably the biggest gap is when you use DRRA, those pods aren’t preemptable. That’s a problem. So, that’s something we’re hoping to address in 137. John Belamaric · ▶ 08:36
we can’t avoid doing this because then Kubernetes as a as a project will not serve all the AI workloads needs that we are seeing. Patrick Ohly · ▶ 24:49
CAP is always a heavyweight thing that if you start working on Kubernetes, you have to design it, you have to defend your decisions. Patrick Ohly · ▶ 13:50
Key Takeaways
- Partitionable devices reached beta in 1.36 after an API redesign removed etcd object-size limits.
- Nvidia donated its GPU and compute-domain DRA drivers; Google donated a TPU driver.
- DRA pod preemption is not yet supported and is targeted for Kubernetes 1.37.
About the Speaker(s)
Patrick Ohly is a Principal Engineer at Intel, member of the Kubernetes Steering Committee, co-chair of WG Device Management, co-chair of WG Structured Logging, and tech lead in SIG Testing. He is the main architect of Dynamic Resource Allocation and has been involved since the initial KEPs.
John Belamaric is a Senior Staff Software Engineer at Google, co-chair of SIG Architecture and WG Device Management. He leads efforts to improve how GPUs, TPUs, NICs, and other devices are selected, shared, and configured in Kubernetes. He is also co-founder of Nephio, an LF project for Kubernetes-based automation.