New Scheduling Features in Karmada v1.14 and v1.15

▶ Watch (2:32)

Karmada releases quarterly. The two most recent versions added four features that fill gaps in multi-cluster scheduling. Version 1.15 extended cluster failover to stateful applications. Before that, failover was disruptive and suited only for stateless workloads. Now a propagation policy can carry a state preservation block with two fields: a JSON path targeting the piece of state to keep, and an alias label name the receiving cluster reads after failover. Version 1.14 added federated resource quota, which enforces CPU and memory limits at the Karmada control-plane level before the scheduler places any workload.

Federated Resource Quota: Keeping Failover Viable

▶ Watch (4:21)

Without capacity guardrails, every member cluster can fill up and failover silently fails. Federated resource quota prevents that. In a concrete example from the talk: two jobs each consuming 10 CPU run against a quota of 20 CPU. One goes to member-1, one to member-2. If member-1 goes down, member-2 has room. Without the quota, teams can keep submitting jobs until all clusters saturate and no failover target has free capacity. The quota syncs to a namespace and the Karmada scheduler checks it before accepting each new resource binding.

Workload Anti-Affinity and Multi-Component CRD Scheduling

▶ Watch (5:34)

Applications sensitive to second-level downtime need replicas spread across data centers. Workload anti-affinity lets operators define logical groups with a label key such as pipeline-id. Karmada places workloads sharing that label value on separate clusters automatically. The inverse, colocation, uses the same mechanism to pin related training jobs to one member cluster. Separately, multi-component CRD scheduling fixes inaccurate resource estimates for CRDs like Flink, where a job manager and a task manager have vastly different CPU requirements. Karmada now extracts per-component requirements into the resource binding before scheduling.

Bloomberg: Model Caching and Jupyter Notebooks Across Clusters

▶ Watch (10:11)

Bloomberg built a model cache API as a thin layer above Karmada. A single request fans out to every node and cluster without custom per-cluster scripts. Propagation policy targets caches by cluster profile, GPU availability, region, or any label. The Karmada API server returns a single status response showing which nodes have downloaded the model and which encountered errors. Bloomberg also schedules Jupyter notebook jobs through Karmada. A notebook requiring V100 GPUs is placed on whichever member cluster has V100s available. Priority-based scheduling is on the roadmap for urgent notebook requests.

Trip.com: 300,000 Pods Migrated and Four-Hour Cluster Onboarding

▶ Watch (15:22)

Trip.com manages more than 40 clusters and some have exceeded 4,500 nodes. Before Karmada, bringing a new cluster online required multiple teams to manually deploy networking, monitoring, logging, and other add-ons, with ordering dependencies that stretched the process to several days. One propagation policy change now rolls all components out automatically in under four hours. For high availability, stateful applications spread across two or more clusters each running independent HPA. When a cluster fails, traffic shifts automatically and HPA scales the receiving cluster. The federation control plane can be unavailable for up to one hour without affecting running services. Cross-cluster migration using propagation policy weight adjustments has moved 300,000 pods across more than 10 clusters without disruption.

Q&A

Does federated resource quota work when the target cluster uses a cluster autoscaler? Karmada sees only a point-in-time snapshot of cluster capacity, so autoscaling capacity that does not yet exist is invisible to the scheduler, meaning some overprovisioning is still needed. ▶ 24:00

Is federated resource quota scoped per namespace and does it cover all resource types? Yes, quota is per namespace and covers all resource specifications. ▶ 25:33

What is the performance ceiling for Karmada at scale? A 2022 benchmark showed support for around 100 clusters. A new performance test is planned for this year after significant improvements made in the past year. ▶ 25:57

Notable Quotes

if all clusters are saturated and one cluster goes down, even if you have cluster failover enabled, those workloads won’t have space to get failed over and rescheduled to the next cluster, and it’s going to make failover useless. Michas Szacillo · ▶ 04:04

this will guarantee that critical workloads are not starved behind a flood of low priority submissions. Tessa Pham · ▶ 10:01

It has reduced the time needed to bring a new cluster online from several days uh to less than four hours. Zach Lee · ▶ 16:35

we have already completed large scale migration across more than 10 clusters involving a total of 300,000 post in a stable way. Zach Lee · ▶ 19:23

Key Takeaways

  • Karmada v1.15 stateful failover preserves state via JSON path expressions in propagation policy.
  • Federated resource quota at the control-plane level keeps failover capacity available across all member clusters.
  • Trip.com cut new cluster onboarding from several days to under four hours across more than 40 clusters.

About the Speaker(s)

Hongcai Ren (@RainbowMango) is a CNCF Ambassador and maintainer of both the Kubernetes and Karmada projects. He has worked on Kubernetes and other CNCF projects since 2019.

Tessa Pham is a Senior Software Engineer at Bloomberg, focused on cloud-native training infrastructure for the company’s Data Science Platform. She previously built Bloomberg’s inference platform.

Michas Szacillo is a senior software engineer and tech lead on Bloomberg’s Streaming Analytics engineering team. The platform runs on Kubernetes and underpins many of Bloomberg’s data streaming use cases. He is a frequent contributor to the CNCF community.

Wei-Cheng Lai is a software engineer on Bloomberg’s AI Platform Engineering team. He builds Kubernetes-based infrastructure for AI applications and contributes to open source projects including Karmada and KubeRay.

Zach Lee (Zongqing Li) is a Senior Cloud-Native R&D Engineer on the container team at Trip.com. His work covers Kubernetes-based HPA, VPA, observability, and multi-cluster management for Trip.com’s large-scale online services.