AI
Why Device Plugins Fell Short
Device plugins date to 2017 and work well in homogeneous clusters where every node looks the same. Real clusters are not like that. Add a second rack with H100s when you already have A100s and the admin must hand-label every node and track which workload holds which device. There is no Kubernetes-native receipt showing “this GPU has been held by pod X since 9 a.m.” Administrators hunting a pod stuck in scheduling for five hours had to log into each node individually to find the answer.
What DRA Adds: Intent, Sharing, and Observability
DRA went GA in Kubernetes 1.35 and introduces three things device plugins never had. First, CEL expressions let a developer request “greater than 24 gigs of memory” instead of “A100 from GPU server04.” Second, consumable capacity lets a 120 GB card serve multiple workloads simultaneously, leaving 70 GB free after three smaller jobs are scheduled. Third, resource claims act as a digital receipt with a status sub-resource that names which pod holds the device and reports driver health events like restarts.
Three Roles, Three Kubernetes Objects
DRA requires three actors to work together. The driver introspects nodes and writes ResourceSlices, the cluster inventory. The admin reads those slices and publishes DeviceClasses, the menu of what users may order. The application developer creates a ResourceClaim that points at a DeviceClass and optionally adds constraints. The scheduler reconciles all three and prevents double-allocation. A ResourceClaim’s lifecycle is tied to the claim object itself, not to the pod, so an allocation can outlive a pod restart and be reused.
Topology Awareness and Consumable Capacity
Two DRA features address expensive hardware waste directly. Consumable capacity lets the scheduler do the arithmetic on VRAM so one GPU card serves several claims at once. Topology awareness prevents a GPU and its NIC from landing on different NUMA nodes, which wastes 10 to 60 percent of throughput via the slow cross-NUMA interconnect. A constraint in the ResourceClaim specifying numa_node on both requests is all it takes to force co-location on the same NUMA node.
The vLLM Custom DRA Driver and Lab Setup
Doug Smith built a custom DRA driver targeting vLLM developers who do not want to write YAML. The driver diffs the working directory, packs the changes, and caches both the diff and the base container image on the cluster. A developer runs kates-has-gpu vlm-run and the driver handles the rest. The lab repo includes Ansible playbooks that spin up a Kubernetes 1.35 cluster with zero physical GPUs required, using emulated devices so all DRA mechanics are visible without real hardware.
Live Workshop: 72 Users, 24 Virtual GPUs
The hands-on portion used three nodes with eight virtual GPUs each, 24 total. Seventy-two attendees SSHed in, ran can-has-gpu to reserve GPUs, and launched vLLM workloads. Load average hit 42 and the cluster slowed significantly. The incident illustrated DRA observability in practice: kubectl get resourceclaims showed every claim, its age, and its current state flipping between allocated and reserved in real time. A 12-day-old claim with a dead pod held its allocation until manually deleted, demonstrating that resource lifecycle is claim-scoped, not pod-scoped.
Notable Quotes
so 95% of your super expensive resource is wasted until that workload is gone Miguel Duarte Barroso · ▶ 04:21
What if I don’t look at the spreadsheet? Miguel Duarte Barroso · ▶ 28:44
72 users and um a load average of 40. Doug Smith · ▶ 01:05:24
once you clean the receipt you will get the device back not when the workload dies. Miguel Duarte Barroso · ▶ 01:11:49
Key Takeaways
- DRA GA in Kubernetes 1.35 replaces manual node labeling with CEL-based intent expressions.
- A single 120 GB GPU can serve multiple claims simultaneously through consumable capacity.
- ResourceClaim lifecycle is decoupled from pod lifecycle, enabling allocation persistence across restarts.
About the Speaker(s)
Doug Smith is a Principal MLOps Engineer at Red Hat, working on the AI Inference Server team and contributing to vLLM through its CI Special Interest Group. He has spent years bridging telecom and cloud-native infrastructure.
Miguel Duarte Barroso is a Principal Software Engineer for OpenShift Virtualization at Red Hat. His work spans SDN, NFV, containers, and virtualization. He maintains the whereabouts and macvtap CNI plugins and contributes to ovn-kubernetes and Multus, and is a member of the Network Plumbing Working Group.