AI
Why AI Clusters Break the Old Kubernetes Scalability Numbers
SIG API Machinery’s official scalability targets sit at 5,000 nodes and 150,000 pods. Production AI training clusters run at 100,000 nodes, a 26x gap. At that scale, pod specs have grown too. Where a typical pod might be 1,000 bytes, an AI workload pod spec is closer to 100 KB. The features covered in this talk address that gap directly, and most of them are invisible to end users unless they notice faster response times or read cloud-provider press releases about giant clusters.
The Three-Request Problem: Why Lists Hit etcd Instead of Cache
The API server holds an in-memory watch cache, but many list requests bypass it and go straight to etcd. Schimanski showed a table from Kubernetes 1.30 where most list variants produced etcd reads. Worse, enabling pagination caused the cache to be skipped entirely. A live demo with 10,000 config maps (each 100 KB) and 20 concurrent workers drove the API server to 18 GB of RAM and generated 63 etcd list calls. That was the baseline this work set out to fix.
The Trilogy: Consistent Reads, Streaming, and Snapshot Paging
Three changes ship together. First, consistent list from cache: instead of going to etcd for a quorum read, the API server asks etcd only for its current resource version, waits up to 3 seconds for the watch cache to reach that watermark, then serves from memory. In the demo, etcd list calls stayed flat at 3 while peak memory dropped to roughly 9 GB. Second, streaming encoding sends each object to the network one at a time rather than building a buffer of up to 2 GB before transmitting. Third, B-tree snapshot paging copies only the nodes on the update path (logarithmic cost) so that a paginated list always reads from the exact tree snapshot taken when the request started, eliminating time-travel bugs where watch events appeared to precede objects already returned.
Watch List: Streaming by Default in 1.35
Watch list reuses the existing watch connection for initial sync. A client adds sendInitialEvents=true to a watch request and receives objects as individual watch events. A bookmark event with a specific annotation signals that the initial list is complete and subsequent events are live changes. In the demo with all four feature gates enabled, memory stayed nearly flat during a 10,000-object list. As of 1.35, client-go enables watch list for informers by default. Updating client-go or controller-runtime is sufficient to participate without any server-side changes.
Controller Sharding: Splitting the Watch Key Space
An alpha feature targets controller scaling directly. Each object gets hashed to a 64-bit integer, and a controller replica requests only a range of that key space. One replica takes the first third of keys, a second takes the middle third, and so on. The API server sends each event only to the replica whose range covers that object’s hash. The long-term goal is to run the Kubernetes controller manager itself in sharded mode. The initial hash target is the object UID, with namespace and node name as candidates for future hash keys.
Q&A
What does a cluster operator need to do to get these improvements? Update to Kubernetes 1.35 and update client-go; streaming encoding and consistent reads are already on by default, and watch list becomes the informer default in client-go for 1.35. ▶ 29:29
Notable Quotes
AI clusters are big, and if you look into the sick API uh the sick scalability mission in the documents in the in in GitHub, you will still find those numbers, right? 5,000 nodes, 150,000 pods and containers. And here you see the comparison with what big clusters in the AI training area like they are 100,000 nodes. Stefan Schimanski · ▶ 09:51
just listed objects and not even so much, right? The number in the AI area with 100,000 nodes is a million pods, not like 10,000. Different dimension. Stefan Schimanski · ▶ 13:06
nothing here is rocket science. So you you will understand in a second what happened. Everything is basically pretty simple changes, but if you see them, you say, of course, why not earlier, right? Stefan Schimanski · ▶ 13:56
memory is not a problem anymore. Stefan Schimanski · ▶ 22:42
Key Takeaways
- Kubernetes 1.35 closes three long-standing list-to-etcd leaks with GA feature gates.
- 20 concurrent quorum-read workers dropped from 63 etcd calls to 3 in the live demo.
- Watch list is now the default informer strategy in client-go for Kubernetes 1.35.
- Controller sharding hashes object UIDs to a 64-bit space, letting replicas split watch load.
- AI cluster pod specs are roughly 100x larger than typical pods, making these fixes urgent.
About the Speaker(s)
Stefan Schimanski is a Principal Engineer at NVIDIA working on control planes, Kubernetes, kcp, and serving as tech lead for SIG API Machinery. He contributed a major part of the CRD feature set and has mentored twice for Google Summer of Code with CNCF.