AI
The Problem: 15 Manual Steps and Weeks of Wait
Before automation, a developer creating a single test cluster walked through 15 manual steps: request capacity, deploy control planes, apply configuration, repeat broken steps. The process took two to three weeks. Uber runs 100-plus production clusters across 15-plus zones, with nodes counts per cluster ranging from 5,000 to 7,000. Over 1,000 engineers want to change those clusters. At that scale, a multi-week manual process for every new cluster is not a staffing inconvenience. It is a hard blocker on shipping new workloads.
Configuration: Static, Dynamic, and Starlark
Cluster config splits into two parts. Static config, the cluster name, zone, and workload type, stays fixed for the cluster’s entire life. Dynamic config changes frequently: node types, controllers, feature flags. Uber uses Starlark, a Python-like language, to merge both layers and produce a final config for each of 200-plus clusters. Engineers configure a cluster type once. Individual clusters inherit that template without needing per-cluster overrides. A compatibility shim converts old config formats to the new model, so existing production clusters migrated without a hard cutover.
Automation: The Kubernetes Cluster Orchestrator
The provisioning system is a Kubernetes-native operator called the Kubernetes Cluster Orchestrator. A developer submits a cluster CRD specifying zone, workload type, and environment. The operator reconciles that object against external APIs: capacity systems, deployment engines, certificate generators, and hub services. Steps are modeled as a graph, with some running in parallel and others gated on predecessors. The operator tracks progress in the CRD status field, reporting which provisioning steps are complete and whether the cluster has reached the active phase. External system failures are handled with retries, rate limits, and cooldowns.
Safe Config Delivery: Sharded Operators and CI/CD
Pushing a config change to all 50 or 100 clusters at once through a single operator is a global blast radius. Instead, Uber shards the orchestrator: one operator instance handles dev clusters, one handles staging, one handles production. Config changes ship as part of the orchestrator binary’s CI/CD pipeline. A developer pushes a controller update to Git. The pipeline applies it to end-to-end test clusters first, runs workload pipelines for batch and stateful jobs, then promotes to staging, then production. No third-party GitOps tool is involved. The same release system Uber uses for all services handles the promotion.
Decommissioning: Safety in Layers
Decommissioning uses the same CRD-and-controller model as provisioning. Three safety layers prevent accidental deletions. First, an admission policy requires a second approver before deletion starts. Second, the operator enforces a quarantine period, giving teams time to cancel. Third, concurrency limits cap how many clusters can be deleted simultaneously. Rate limiting toward downstream systems also blocks mass actions like releasing thousands of hosts at once. The operator stores a full audit history in the CRD status field. A mistake during decommissioning can cause data loss or a production outage, so each layer is independent.
Q&A
Do OS and Kubernetes version upgrades go through this same process? Version upgrades are handled by a separate subsystem owned by the same Compute team, and they are also automated. ▶ 27:04
Why build a custom operator instead of using Cluster API or vcluster? Uber evaluated both but found they could not integrate cleanly with Uber’s internal capacity, networking, and deployment systems. ▶ 28:34
How long did it take and how many engineers built this? The core automation and config layer took about six months to reach production readiness. One team of roughly four engineers built it. ▶ 35:55
Do you need a full-scale duplicate cluster to test changes before production? No. Benchmark clusters simulate production-scale load, including 5,000-node behavior, without requiring a full mirror of every production cluster. ▶ 32:03
Notable Quotes
cluster provisioning require you to go through the 15 manual steps as a developer and it doesn’t sounds really cool right also it’s h gives you a lot of uh weeks of work because it’s really really hard you don’t even have single source of truth when you as a developer as a cluster owner can see what’s goal state going to be for your cluster Vadim Plakhtinskii · ▶ 04:47
it only takes hours instead of weeks. Silvio Simunic · ▶ 25:13
if you break something it will be like disaster right you people can order food and stuff so for us it was more easy to create our own stuff then adopt already existing things Vadim Plakhtinskii · ▶ 31:35
we don’t want only one operator run and operates but we want to shard it and say okay we have one operator which will serve only def clusters another one which will handle staging and final one with canal production Vadim Plakhtinskii · ▶ 20:35
Key Takeaways
- Provisioning 5,000-node clusters dropped from two to three weeks to hours after automation.
- Starlark merges static and dynamic config, giving 200-plus clusters a single source of truth.
- Sharding the operator by tier (dev, staging, production) turns config rollouts into standard CI/CD promotions.
About the Speaker(s)
Silvio Simunic is a software engineer on Uber’s Compute team, focused on Kubernetes infrastructure, cluster lifecycle management, and orchestration.
Vadim Plakhtinskii is a senior software engineer with deep focus on Kubernetes, distributed systems, and formal verification using TLA+. His work spans cloud-native Kubernetes infrastructure design and large-scale data platforms.