AI
Kubernetes Scalability Is a Multi-Dimensional Problem
Node count is one dimension of Kubernetes scalability, not the definition. SIG Scalability tracks dozens of dimensions: nodes, pod count, pod churn, number of gateways, number of secrets, and more. The SIG defines a “scalability envelope,” a zone within which all scalability SLOs are satisfied simultaneously. Approximating that envelope by treating dimensions as independent works near the center but breaks near the boundaries. Any production deployment approaching those boundaries needs independent validation before going live.
Two SLOs, Measured on Real Clusters Across AWS and GCP
The two mature SLOs are API call latency and pod startup time. Definitions written in 2015 were coarse enough that the same system could be read as passing or failing depending on interpretation. SIG Scalability tightened those definitions and now gates releases on them. Tests run on real clusters, not simulations. In February, a 5,000-node test showed AWS green and GCP red. The regression was in kube-proxy, a component AWS replaces with a different stack. Running both providers in parallel both caught the bug and narrowed the diagnosis immediately.
The Cluster Loader and PerfDash Testing Stack
Cluster Loader is a semi-declarative framework. You define desired cluster state, and the framework drives the cluster toward it while measuring SLOs and dumping profiling data. Grafana and Prometheus integration is built in. PerfDash sits on top and tracks metric history across runs, making regressions visible as a trend rather than a single-point failure. Both SLOs are measured automatically in every test case, including custom ones written by feature owners. SIG Scalability asks feature owners to write or extend scalability tests themselves; the SIG cannot track every new feature in the community.
WatchList and Streaming Serialization: Fixing API Server Memory Spikes
A single list request can exceed 1 GB of data. The API server serializes the entire response in memory before sending it, causing unpredictable spikes. WatchList, started in 1.27 or 1.28, replaces list-then-watch with a single watch call that streams the initial state in chunks, capping memory per request at roughly 2 MB. In 1.35, client-go switched to WatchList by default. Separately, JSON and protobuf serialization now streams individual list items instead of buffering the full response, achieving the same memory benefit for clients that still issue regular list requests.
Snapshot-Table Watch Cache: Removing etcd from the List Path
Before 1.32, every consistent list request hit etcd, which is frequently the bottleneck. Consistent Lists from Cache (introduced in 1.32) added a mechanism where the watch cache asks etcd only for the current resource version, then waits until its in-memory state catches up before serving the response. The watch cache now stores objects in a B-tree structure that supports copy-on-write snapshots. This lets paginated requests read a snapshot from seconds ago while newer requests read fresher data concurrently. Today, every list request, paginated or not, consistent or not, is served from the API server cache.
Opportunistic Pod-Signature Batching and Large-Object Work
The scheduler now assigns a signature to pods that are identical from a scheduling perspective: same requirements, same shape, same node constraints. Feasibility computation done for the first pod is reused for subsequent pods with the same signature. The primary target is one-pod-per-node AI and ML workloads. This raises the pod churn ceiling rather than reducing memory. A second active area is large-object handling. Tests passed with synthetic payloads, then failed for real users whose objects were simply larger. Initial fixes land in 1.36, but Tyczyński described this as work that started roughly two months ago with significant effort still ahead.
Notable Quotes
the the amount of data that we need to send in the single list request can even can easily exceed like a gigabyte of data in a single request Wojciech Tyczyński · ▶ 17:03
it pretty much every single list request no matter if it’s like pagionated, non-pagenated, consistent, non-consistent, whatever, it’s already self served from um from API server Wojciech Tyczyński · ▶ 25:44
they were using larger objects, not more of them, not faster, not whatever, but they were just larger Wojciech Tyczyński · ▶ 31:13
Key Takeaways
- WatchList caps API server memory per request at roughly 2 MB, replacing unbounded list serialization.
- Running release-blocking tests on both AWS and GCP caught a kube-proxy regression that AWS alone would have missed.
- Every list request now serves from the API server watch cache; etcd only feeds that cache.
About the Speaker(s)
Wojciech Tyczyński is a Senior Staff Software Engineer at Google, working on Google Technical Infrastructure and Cloud since 2012. Since 2015 he has focused on Kubernetes and GKE, contributing to scalability, performance, and reliability across most Kubernetes components. He holds an M.S. in Computer Science and a B.S. in Mathematics from the University of Warsaw.