AI
Kubernetes Job API: Four Features Graduating in 1.34
Pod replacement policy reaches GA in Kubernetes 1.34. It controls whether a job creates replacement pods while previous pods are still terminating. The default strategy, “terminating or failed,” preserves the old behavior. The new “failed” strategy blocks replacement pod creation until prior pods are fully gone, regardless of whether they are active or terminating. Two other features also graduate: mutable pod resources for suspended jobs, letting controllers change resource requests mid-suspension, and mutable scheduling directives, which cut job-resume API calls from two down to one.
JobSet: Ordering, Persistent Volumes, and In-Place Restart
JobSet now supports a depends-on field that enforces execution ordering across replicated jobs. The Kubeflow community requested this to sequence data download, model pull, and training into a single declaration. Volume claim policies let jobs attach storage that outlives the workload, useful for checkpoints, while temporary scratch volumes are cleaned up automatically. The most involved addition is in-place restart, built on a new pod API called “restarts in place.” At 10,000-node scale, recreating every pod for one failure is not viable, so this lets a single failing job reschedule without touching the rest.
Kueue Topology-Aware Scheduling: Three Algorithms Compared
Kueue ships three topology scheduling algorithms selectable via pod template annotations and feature gates. Best-fit greedily assigns pods to the largest free topology domain first. Rack-free capacity mirrors Kubernetes bin-packing, packing all pods into the fewest racks possible. Balance placement, available only in preferred mode, spreads a minimum pod set across each rack, which matters for NVLink hardware like the GB200 and GB300 where interconnect NIC count depends on GPUs present. Multi-layer topology adds a hard constraint: the scheduler refuses assignment unless replica count divides evenly across remaining domain nodes.
Kueue 0.15 to 0.16: API Migration and Upcoming 0.17
Kueue promoted its core API from v1beta1 to v1beta2 across two releases. Version 0.15 introduced v1beta2 with a conversion mechanism while keeping v1beta1 as storage. Version 0.16 switched the storage version to v1beta2. Version 0.18 will stop serving v1beta1 entirely. Users still on v1beta1 should migrate now. Kueue 0.17, planned to release the following Monday or Tuesday, adds DRA extended resource support, elastic job support, multi-area topology constraints, StatefulSet and serving workload support in MultiKueue, and atomic preemption across cluster admissions to prevent preemption collisions between clusters.
Gang Scheduling Enters Core Kubernetes via Workload-Aware Scheduling
Gang scheduling requirements differ sharply by workload type. A plain Job wants all-or-nothing. A JobSet has a leader that must schedule before workers can start, and leader preemption can kill the whole job. Disaggregated serving adds a required ratio between prefill leaders and decoders sharing a KV cache. To handle all of these, WG-Batch introduced two new APIs targeting Kubernetes 1.36: Workload, a static definition with a minimum pod count, and PodGroup, a runtime object with status. A job links to a PodGroup template and the scheduler treats the group atomically. Six features shipped for this area, with some merging two days before the conference.
Live Demo: Bring-Your-Own PodGroup and Gang Job
Kevin ran two demos against a local cluster built from main. The first created a Workload and a PodGroup pointing to a template with 1,000 pods, then created a Job linking to that PodGroup. All 1,000 pods entered pending, proving the scheduler treats the group as a unit before placing any of them. The second demo used a gang-job manifest where the batch Job controller creates the PodGroup and Workload automatically. The PodGroup status showed pending briefly before the scheduler placed the group, demonstrating that controller-level integration works without manual PodGroup management.
Q&A
What is the Slack channel for workload-aware scheduling? The channel is named “workload-aware-scheduling” in the Kubernetes Slack workspace. ▶ 31:29
Notable Quotes
my favorite part about working group batch has always been that our goal was always to contribute directly to core kubernetes to improve the use case for all batch workloads. Kevin Hannon · ▶ 09:46
I did not realize how hard it is to honestly run workloads on GPU nodes. There’s always a lot of failures. Kevin Hannon · ▶ 08:41
I don’t think anyone should ever worry about workloads or pod groups. If you come from the a HPC world they don’t think about gang scheduling. It’s just done for you. Kevin Hannon · ▶ 27:15
I’m feeling the the fact that we don’t have a true gang scheduling solution in Kubernetes and it makes supporting AI workloads really challenging. Kevin Hannon · ▶ 31:10
Key Takeaways
- Pod replacement policy reaches GA in Kubernetes 1.34 with a new “failed” strategy that blocks premature pod recreation.
- JobSet’s in-place restart uses a new pod API to avoid full workload teardown on 10,000-node clusters.
- Kueue 0.18 will drop v1beta1 API serving; migrations from v1beta1 to v1beta2 should happen now.
- Workload and PodGroup APIs bring native gang scheduling to Kubernetes 1.36 across Job, JobSet, and Kueue.
- Disaggregated serving workloads require enforcing prefill-to-decoder ratios, a scheduling problem WG-Batch is still actively specifying.
About the Speaker(s)
Kevin Hannon is a Senior Software Engineer at Red Hat focused on improving HPC, AI, and ML workloads on Kubernetes. He started his career as a computational scientist and has maintained a focus on large-scale batch jobs throughout. He is a co-chair of WG-Batch.
Yuki Iwai is a Software Engineer at CyberAgent, Inc., where he works on the internal platform for machine-learning and high-performance computing. He is Lead for Kubernetes WG-Batch and Kubeflow WG-Training, and a reviewer for kube-controller-manager Job code.