AI
Why Kubernetes Struggles Across Multiple Clusters
Kubernetes was built on the assumption that the cluster is the universe. For most of its life, nothing outside a cluster mattered. That meant clusters had no self-reference, no way to be addressed externally, and no concept of peer clusters. Multi-cloud deployments, compliance isolation, canary clusters, and latency-driven regional placement all break that assumption. SIG-Multicluster exists to fix it, with APIs that work across existing tools rather than replacing them.
Six Building Blocks: From Cluster Profile to Multicluster Runtime
Stephen Kitt walked through all six projects. Cluster Profile stores names, credentials, and hardware properties (such as GPU availability) on the management cluster. A companion API tells each member cluster its own name and cluster-set membership. Work API pushes resources from the management cluster out to members. Placement Decision records where workloads should go. MCS API exports a service from one cluster so workloads in another cluster can reach it without a manual gateway. Multicluster Runtime is a controller-runtime equivalent for watching objects across an entire cluster set.
Placement Decision: Decoupling Scheduling from Deployment
The Placement Decision API separates two concerns that previously had to be bundled together. A placement controller reads requirements, such as a PyTorch training job that needs eight GPUs, queries Cluster Profile for available clusters, and writes its choice into a Placement Decision object. A separate syncer reads that decision and deploys the workload. The syncer never needs to understand GPU constraints. For progressive rollouts, the same object gets updated from 10% and 25% clusters to 75%, and the syncer reconciles the difference without any change to its logic.
User Research: What Multi-Cloud Operators Actually Hit
Starting last year, with community member Priya driving the work, the SIG ran user surveys and built a primary persona named Alex. Alex is a hybrid engineer managing clusters across on-prem, edge, and at least two cloud providers. The top pain points from the research: documentation gaps make it hard to know where to start, observability consistency is broken across cloud providers, and operational complexity surfaces late rather than upfront. AI-driven automation for multicluster scored high interest but depends on fixing documentation and observability first.
Q&A
Is there a plan for a Placement Request API to complement Placement Decision? The SIG is still debating whether one is needed, and no formal proposal has been submitted yet. ▶ 24:21
Does syncing member-cluster state back to the hub cluster cause performance problems? Bulk state syncing back to the hub creates single-cluster scaling limits, so the SIG discourages it and recommends keeping the hub as an abstract representation with relatively static status. ▶ 28:15
Should MCS API track changes to Service and Gateway API? The SIG’s position is that MCS should match Service: when Service changes, MCS grows with it, so operators do not have to relearn concepts when moving to multicluster. ▶ 30:18
Notable Quotes
multicluster is everywhere but there’s a lot of different reasons uh why you adopt multicluster right you could have footprints in multiple clouds you could be running across multiple regions uh it could be for isolation between you know various classes of applications could be compliance could be for release management Jeremy Olmsted-Thompson · ▶ 01:02
the scheduleuler understands GPU requirements but the consumer doesn’t need to know anything about that. It just is able to read this and place things Stephen Kitt · ▶ 11:53
it’s like this gradually discovered uh sort of latebinding toil um where the differences kind of rear their head and you want to actually understand how how you can get a better upfront view of that consistency Jeremy Olmsted-Thompson · ▶ 19:10
half the time I don’t say anything. Um, but I just listen in because it means that you also get this um this option where you might think you have problem A, but it turns out you actually have problem B, but you never heard problem B described, so you didn’t know you have it. Community member · ▶ 23:43
Key Takeaways
- Placement Decision API decouples GPU-aware scheduling from deployment, letting syncers act without knowing constraints.
- Cluster Profile stores per-cluster hardware properties but syncing dynamic data like free memory back to the hub kills performance.
- SIG user research named documentation gaps and cross-cloud observability inconsistency as the top blockers for multi-cloud operators.
About the Speaker(s)
Stephen Kitt is a Senior Principal Software Engineer at Red Hat and a maintainer of the Submariner project, which provides connectivity and service discovery across multiple Kubernetes clusters. He has been at Red Hat since 2015, working on OpenDaylight and Submariner.
Jeremy Olmsted-Thompson is a Principal Engineer at Google working on Google Kubernetes Engine. His focus is simplifying the Kubernetes experience, including cross-cluster deployments with multi-cluster solutions.
Laura Lorenz is a Software Engineer at Google and an active contributor to Kubernetes upstream, with focus areas in SIG-Multicluster, SIG-Node, and releases. She contributed substantially to the slide content for this session.