AI
Why Monolithic AI Jobs Waste GPU Capacity
Enterprises schedule AI training jobs as a single monolith, placing the entire job in one cluster to guarantee network performance. Resource fragmentation is unavoidable as jobs run day to day. Bin-packing and priority preemption help but are not silver bullets. The result is resource bubbles: expensive GPU capacity sits idle because the cluster cannot fit the whole job. Clusters are built with careful boundaries, then teams spend even more engineering effort trying to cross those same boundaries.
Pipeline Parallelism as the Cross-Cluster Starting Point
Tensor parallelism demands extreme network bandwidth and must stay inside a single node or super-pod. Data parallelism requires massive gradient synchronization across clusters, making it impractical without a very high-bandwidth, low-latency interconnect. Pipeline parallelism splits the model layer by layer into stages and passes only lightweight activations and gradients across the network boundary. That lighter communication makes it the most practical starting point for cross-cluster training today, without requiring a specialized multi-cluster network fabric.
The Asymmetric Layer Split: Eliminating Idle Bubbles
A symmetric 50/50 split between a B200 cluster in Beijing and an H100 cluster in Shanghai creates a visible problem. Each cluster receives four layers. The B200 finishes its computation instantly and then stalls waiting for Shanghai. Expensive B200 compute cycles burn on nothing. The fix is asymmetric: assign six layers to the faster Beijing cluster and two layers to the slower Shanghai cluster. Throughput per micro-batch step aligns, the idle bubble disappears, and the zero-bubble optimization becomes applicable.
HyperJob: A Federation-Native API for Job Decomposition
The HyperJob API sits above existing Volcano jobs. A user declares the model, data location, and container image once. The HyperJob controller splits that declaration into multiple Volcano jobs, generates a Karmada propagation policy for each, and adds scheduling constraints that pin related jobs to clusters with compatible network proximity. Volcano Global handles global queue management and priority ordering across tenants. Gang scheduling at the HyperJob level is planned. The current splitting policy is static and predeclared, with automated partitioning on the roadmap.
Accurate Resource Estimation Across Member Clusters
Karmada previously estimated workload size by taking the largest resource template and multiplying by total replicas. A PyTorch job with separate master and worker nodes has different resource requirements per template, making that estimate inaccurate. Recent releases added a resource estimator that collaborates with the Karmada scheduler to calculate per-template requirements precisely. The scheduler can now identify which member cluster can actually run the job and how much capacity remains, enabling bin-packing and idle-cluster-first scheduling policies.
What Comes Next for Volcano Global
The near-term priority is a smarter splitting policy with dynamic weighting based on real cluster resource awareness. Data parallelism across clusters is also in progress. The plan is to optimize the all-reduce operation hierarchically to cut cross-cluster data exchange and reach acceptable end-to-end performance. Karmada’s existing scheduling groups feature, which lets users declare multiple fallback scheduling preferences, will integrate with the HyperJob partitioner so the system can find a valid placement more easily when first-choice clusters are unavailable.
Notable Quotes
the the pipeline uh parallelism splits the model vertically, right? uh layer by layer uh into different uh stages. And the the the brilliant brilliant part is is the boundary because only it only passes a relatively lightweight activations uh and gradients across the network instead of instead of syncing the entire model Kevin Wang · ▶ 08:52
you your you your kind of running faster cluster are uh you know uh literally uh literally uh stalling waiting for the slow cluster to finish their forward and then you know back backward passes you are kind of you know uh burning incredibly incredibly uh expensive uh B200 compute cycles on absolut absolutely nothing Kevin Wang · ▶ 10:26
it’s kind of a funny fact that we you know spend massive engineering effort to to build our cluster with some of the boundaries we mentioned above but uh then we spend even more effort trying to you know cross this boundary Kevin Wang · ▶ 03:15
Key Takeaways
- Monolithic job scheduling leaves expensive GPU clusters idle due to unavoidable resource fragmentation.
- Pipeline parallelism with asymmetric layer assignment eliminates idle bubbles between heterogeneous clusters.
- The HyperJob API, already released, automates Volcano job decomposition and Karmada propagation policy generation.
About the Speaker(s)
Kevin Wang is a Technical Expert at Huawei and Vice Chair of the CNCF TOC. He has contributed to Kubernetes since the community’s early days and leads Huawei’s cloud native open source team. He initiated and led the incubation of KubeEdge, Volcano, and Karmada inside CNCF.