AI
When Vertical Scaling Hits a Ceiling
LY Corporation’s PaaS runs 66,000 pods across 5,000 nodes, 3,000 namespaces, and 500,000 managed objects. At that scale, the team started seeing slow initial synchronization during restarts, longer reconcile latency, and delayed visibility for users. The system was not failing, but it was approaching a structural limit. Standard Kubernetes controllers use a shared informer cache with a single active reader. Leader election means only one replica reconciles at a time. Adding more replicas does not increase throughput. Vertical scaling was the only lever, and it would not hold forever.
How KCS Assigns Shard Ownership
Kubernetes Controller Sharding (KCS), an open-source project on GitHub, makes controllers horizontally scalable. It computes a hash of each resource’s identity using XXhash, then uses binary search on a consistent hashing ring to assign ownership. Each shard gets 500 virtual nodes distributed across the ring. For main resources, the key includes group, kind, namespace, and name. For child resources, the key uses the parent’s name, so parent and child always land on the same shard. KCS handles ownership assignment. It does not handle arbitrary dependency graphs or act as a distributed cache framework.
The Cost of Namespace-Level Sharding
The team’s first design started a controller-runtime manager for each namespace, so each shard would handle only the namespaces assigned to it. On paper, this looked cheaper than one pod per namespace. In practice it was not. The in-process manager instantiation cost matched the pod-per-namespace cost. Before sharding, the operator ran about 40 manager instances, one per data plane cluster. After namespace sharding, each of those managers replicated across 3,000 namespaces: 3,000 times 40 equals 120,000 manager instances. Goroutines climbed from 3,500 to over 500,000. The pods hit OOM loops.
Splitting the Operator to Bound Manager Count
The team split the monolithic operator into an app operator and a deploy operator. App operator uses label-based sharding: one manager per pod, all namespaces watched through a single informer set. Deploy operator shards by data plane cluster: one pod per cluster, one base manager plus one cluster manager per pod. Before the pivot, one pod held 40 cluster managers, caches, and goroutines for all clusters. After, each pod held one. Manager count per pod is fixed. Cost is bounded and blast radius is limited to one cluster. Reconcile latency dropped. The system has been stable in production since.
Two Production Edge Cases at Scale
Two edge cases appeared once the system ran at scale. First: resources created without owner references are invisible to sharded controllers. A precreated app child had no owner reference, so no shard was assigned and no controller reconciled it. The fix was adoption logic that queries the API server directly and sets the owner reference, letting KCS assign the correct shard. Second: deploy used a Deployment, so rolling updates generated new random pod names. Shard identity changed. All 11,000 managed objects needed relabels at once, overloading the API server. Switching to a StatefulSet gave stable names and solved it.
Notable Quotes
amplified 3,000 times was catastrophic. Motohiro Otsuka · ▶ 13:49
It is not a distributed cache framework. Motohiro Otsuka · ▶ 16:15
Predictability is a form of scalability. Motohiro Otsuka · ▶ 20:18
Shading is not a feature toggle. Tomoyuki Nakamura · ▶ 27:32
Key Takeaways
- Namespace sharding amplified LY’s operator from 40 manager instances to 120,000, triggering OOM.
- Keep manager count per pod fixed: never let it scale with an external dimension like namespaces.
- KCS assigns ownership along parent-child chains; it does not partition arbitrary dependency graphs.
- Use StatefulSet for sharded operators: Deployment rolling updates trigger mass relabeling at 11,000 objects.
- The right sharding axis depends on which dimension (namespaces or clusters) dominates your architecture.
About the Speakers
Motohiro Otsuka is a platform engineer at LY Corporation, building the company’s internal Kubernetes-based platform. Previously, he was an OpenStack core reviewer and OSS contributor. His current interests center on Kubernetes controllers, multi-cluster operations, and cloud-native infrastructure.
Tomoyuki Nakamura is a platform engineer at LY Corporation. He joined Yahoo! Japan in 2019 and moved to LY Corporation in 2023 after the company merger.