AI
What Slurm Is and Why It Still Runs Most AI Training
Slurm is the workload manager behind the majority of the Top 500 supercomputers and most AI/ML training clusters. It prioritizes jobs, decides which jobs run on which nodes, tracks node state, and assigns resources. A single Slurm cluster scales beyond 15,000 nodes. About 80% of the KubeCon audience already knew it. For the remaining 20%: think of it as a scheduler and resource manager rolled into one, licensed under GPLv2, the same license as the Linux kernel.
How Slurm Bridge Turns Slurm into a Kubernetes Scheduler Backend
Slurm Bridge plugs the Slurm controller into the Kubernetes scheduler framework as a backend. When a pod is submitted, the bridge translates its resource requests into a Slurm placeholder job. Slurm decides when and where the pod runs, then writes node assignments back as annotations. Kubelet handles the actual launch. Before this design, a node had to be marked either Slurm or Kubernetes. Running both meant picking one scheduler per node and leaving the other idle.
Why Kubelet CPU Management Falls Short for HPC Workloads
Kubelet’s static CPU policy pins CPUs to a workload, but it treats all CPUs as identical. It has no awareness of NUMA topology, L3 cache boundaries, or socket layout. A Slurm job that needs five performance cores on a specific NUMA node cannot express that request through kubelet today. The CPU DRA driver fills this gap. It uses Dynamic Resource Allocation, GA since Kubernetes 1.34, to let workloads request CPUs by attributes: core type, socket, NUMA domain, or L3 cache group.
Two Resource Representation Modes in the CPU DRA Driver
The CPU DRA driver publishes node topology in two modes. Individual mode lists every CPU as a separate device, CPU 0 through 48, each tagged with core ID, L3 cache, and socket. Slurm Bridge uses this mode: Slurm picks exact CPUs and injects them into the pod as a resource claim. Group mode pools CPUs into logical blocks by NUMA, socket, or core type, then uses DRA’s consumable capacity feature to draw an arbitrary count from a pool. Each socket advertises 32 CPUs in the demo cluster.
What DRA Unlocks: Mixed Workloads on the Same Node
With DRA managing all resource assignment, kubelet owns every node. Slurmd no longer needs to run everywhere. Slurm added external node support in its latest release so the controller can schedule to nodes it does not manage directly. The result: each node runs Slurm and Kubernetes workloads at the same time. Three DRA drivers are supported today: the CPU DRA driver, a GPU example driver used for testing, and the NVIDIA GPU driver (alpha). Multiple pods per node is also alpha.
Live Demo: CPUs and GPUs Assigned Through a Single Resource Claim
The demo ran on a nine-node kind cluster with the CPU DRA driver and GPU example driver installed. A pod requested both a CPU and a GPU. Slurm scheduled the job, created a resource claim with exact CPU and GPU assignments, and injected it into the pod spec. The NRI plugin enforced the CPU pinning through cgroups and removed those CPUs from the shared pool for other containers. The pod moved from pending to running in under two minutes, with CPUs and GPUs both visible in the resource claim.
Q&A
Does Slurm Bridge still require allocating the whole node? Whole-node exclusive allocation has been turned off, though extensive testing has not been completed. ▶ 24:56
Is Slurm now a first-class citizen in Kubernetes? Networking alignment is the remaining gap; the other pieces are in place. ▶ 22:42
Can OpenMP workloads inside a container see only their pinned CPUs? The container sees cgroup enforcement but can still read full CPU info from the OS; a known workaround exists but the exact mechanism was not recalled on stage. ▶ 21:08
Has the CPU DRA driver been released? The last PR was merging at time of the talk, with a release expected the same day. ▶ 25:59
Notable Quotes
Slurm is a leading high performance computing workload manager. Marlow Warnicke · ▶ 00:26
each node can run both slurm and kubernetes workloads simultaneously. So this makes a big difference. So you no longer have to say is it slurm is it kubernetes I don’t know. You don’t have to choose. Marlow Warnicke · ▶ 13:41
the DRA driver just like respects whatever is there in the resource claim and through NRI actuates them on the at the croup level. Praveen Krishna · ▶ 21:19
So now with dr we can schedule both kubernetes and slum job without put it slurm into maintenance mode. Correct. Correct. Yeah. Audience / Marlow Warnicke · ▶ 25:04
Key Takeaways
- Slurm manages most AI/ML training clusters and scales beyond 15,000 nodes.
- DRA’s consumable capacity lets workloads request CPUs by NUMA, socket, or core type.
- Each node can now run Slurm and Kubernetes jobs at the same time, without choosing one scheduler.
About the Speaker(s)
Marlow Warnicke is a Senior Engineer at Nvidia and chair of the CNCF Batch Working Group. Her background spans resource management, HPC system tools, kernel drivers, tracing libraries, and security. She leads the Slinky project.
Praveen Krishna is a Software Engineer at Google with experience in large-scale cluster orchestrators from Borg to Kubernetes. He focuses on Kubernetes resource management for HPC workloads and drove the CPU DRA driver work described in this talk.