AI
Why GPU Scheduling Fails and How to Know It
Kueue is a batch job dispatcher for multi-tenant GPU clusters where compute is expensive and organizations target close to 100% utilization around the clock. Large organizations assign GPUs to teams through cost centers and quotas. When a tenant submits workloads that should fit within their guaranteed quota but stay pending, that is a guarantee violation. It is the first signal that something has gone wrong, and it is the hardest question any batch scheduler operator faces, whether running Kueue, Slurm, or Volcano.
Why the Scheduling Problem is Computationally Hard
Even a simplified three-processor makespan problem is NP-complete. Kueue’s problem is far harder: thousands of non-identical machines, a dynamic node set, and an online stream of workloads. Kueue must enforce quotas, prioritize high-priority jobs, maximize utilization, avoid unnecessary preemptions, and maintain fair sharing simultaneously. To stay practical, Kueue uses heuristics including priority queues, sorting, scoring, and fair sharing rather than searching for an optimal solution.
A Six-Step Recipe for Diagnosing Scheduling Failures
The recipe starts with detecting a guarantee violation: compare allocated quota, nominal quota, and pending workload quota for each cluster queue. Step two rules out fragmentation by counting available node slots that match the pending workload shape. For example, twelve 8-GPU workloads need at least one free 8-GPU-shaped slot per node. Steps three and four check admission and preemption rates, then confirm whether Kueue has released the workload to Kubernetes via scheduling gate removal or unsuspend. Steps five and six scan logs at zap level five and check for system health changes.
Bug 1: Head-of-Queue Blocking with 15-Minute Preemption Delays
The root cause was head-of-queue blocking when using the pod workload type. Kueue waited for preemption targets to finish terminating before admitting the next candidate. Users had configured a pre-stop lifecycle hook and grace period termination of 15 minutes, so only one workload was admitted to the demo cluster queue every 15 minutes. This surfaced during cluster migrations and during spikes of high-priority inference workloads. The fix, merged upstream, releases quota for a pod workload type immediately once it is marked as a preemption target, matching the behavior of the job workload type.
Bug 2: Zombie Pod Blocking Admission Across All Cluster Queues
A large zombie pod became a preemption target for heads across multiple cluster queues. Because Kueue skips a cluster queue whose head shares a preemption target with another already-processing workload, admission slowed across the entire cluster. The zombie pod appeared when the kubelet failed or when a pod was terminated mid-image-pull. The temporary fix was a forced pod deletion. Kueue now releases quota for preempting pods that exceed their grace period termination time, so a single stuck pod no longer halts the full scheduling cycle.
Bug 3: Rounding Error in Dominant Resource Share Values
This bug only appeared after enabling preemption fair sharing. Guarantee violations coincided with that configuration change. Raising the zap log level from two to five exposed odd dominant resource share values during the admission tournament. A rounding error assigned a DRS value of zero to cluster queues that were actually borrowing beyond their guarantees. Kueue treated those borrowing queues as within-guarantee, leaving no valid preemption targets for queues legitimately reclaiming their quota. The fix is merged upstream, credited to contributor Dominic. Operators on current Kueue versions will not encounter this.
Q&A
How should operators track pending workload resources before issue 9741 lands? Create a custom metric in kube-state-metrics that fires when a workload requests capacity but its QuotaReserved and Admitted conditions are both false. ▶ 17:00
Notable Quotes
compute is expensive and researchers need to split up GPU compute time in a reasonable way. Amy Chen · ▶ 00:42
one workload is being admitted to the demo cluster Q every 15 minutes Amy Chen · ▶ 18:34
the root cause was a stuck zombie pod. The temporary mitigation was to forcibly delete the zombie pod. Amy Chen · ▶ 26:03
it turned out there was a rounding error where in some cases we assigned a DRS value of zero to cluster Q’s which were actually borrowing. Gabe Saba · ▶ 28:16
tracking guarantee violations can initially alert you to an issue before your researcher’s notice. Amy Chen · ▶ 29:22
Key Takeaways
- A guarantee violation, pending workloads that fit nominal quota but do not admit, is the primary signal of a scheduling failure.
- Three production bugs each caused multi-minute or multi-hour admission stalls and are now fixed in upstream Kueue.
- Two new metrics, issues 9741 and 7539, will expose pending resources per cluster queue and cohort tree hierarchy directly.
About the Speaker(s)
Amy Chen is a software engineer at CoreWeave, where she builds the platform running CoreWeave’s ML workloads. Her entire career has been built on Kubernetes, with prior roles at Apple, VMware, Heptio, and Rancher Labs. She is also an upstream Kueue contributor.
Gabe Saba is a software engineer at Google and a maintainer of Kueue. He previously worked on Amazon Alexa’s Machine Learning Platform and the Google Travel team, and now focuses on improving the ML training experience on Kubernetes.