AI
Uber’s Stateless Compute Stack at Scale
Uber’s stateless compute platform sits across more than 50 Kubernetes clusters spanning multiple regions, ingesting hardware from GCP, Oracle, and on-premises. The federation layer, called UP, is the interface service owners use to deploy and scale services daily. UP places workloads into zones, reads a status object from an in-house CRD called UB Deployment, and drives an internal state machine that triggers the next build when status reports success. The team manages over 10 million containers with one and a half pod launches per second on average.
Why a Second Orchestrator Created a Conflict
Uber operates two data centers in active-active mode, each carrying roughly 50% of traffic. A regional failover drains all traffic from one region and redirects it to the other. Historically Uber kept double the CPU and memory capacity in the standby region just to absorb that load. When engineers decided to use that idle capacity for low-tier workloads and evict them during failover, a second orchestrator appeared alongside UP. Two orchestrators now needed to agree on replica counts for the same services, and UB Deployment Controller was not built for that.
The ServiceScale CRD: One Place to See All Intentions
The team spent six to ten weeks designing a new CRD called ServiceScale. It acts as a sidecar to each UB Deployment object and holds a map of desired replica counts from every orchestrator. When UP wants 10 replicas and the failover orchestrator wants zero, both intentions sit in one object and a simple conditional resolves them. Failback requires no external API calls and no database reads because the previous intention is already materialized in the spec. That auditability was the primary reason the team chose this design over more complex approaches.
The Consistency Trap in Status Computation
The ServiceScale status field exposed a deeper problem. Controllers read objects from informer caches that can lag by several seconds. If the cache returns a stale view and the controller reports workflow status as success, UP’s state machine advances to the next build with no way to reverse. To guard against this, the team built a “read your own right” guardrail. The UB Deployment Controller stamps its own generation as an annotation on downstream objects and refuses to compute a success status until that generation appears in its reads.
Stuck Workloads: Detection, Healing, and an Upstream PR
Near-simultaneous writes from two controllers, combined with stale informer caches, caused annotation-to-spec drift on ReplicaSets. The drift broke proportional scaling and left tier-0 and tier-1 services stuck in production with no automated recovery path. The team built a cluster inspection automation that detects the drift condition and emits alerts, then added a healing job that patches the affected ReplicaSet annotation back in line with the spec. A fix to the upstream Kubernetes deployment controller’s fast-scale path is in a PR, but patching controller-manager in production was not an option while services were stuck.
Integration Tests as the Deployment Safety Net
The team had no outage budget for the rollout. Every line of the ServiceScale controller carried at least three or four unit tests. Race conditions from the read-your-own-right guardrail cannot be reproduced in unit tests alone, so the team ran a kind cluster setup to simulate concurrent writes. That setup caught issues that controller-runtime fakes would have missed. The project shipped without user-visible downtime across all Uber services. The lesson: integration tests against lightweight Kubernetes clusters are worth the maintenance cost when the blast radius is this large.
Notable Quotes
u that’s why the council of did not give the ring to barome you don’t give more power to the already powerful. Srikar Paruchuru · ▶ 10:56
we do have more than 50 clusters running in production and I know for sure that we do one and a half pot launches per day. Egor Grishechko · ▶ 08:38
it was the first piece of software in my life that each line was covered at least three or four unit tests. Egor Grishechko · ▶ 22:50
there’s no final state. Recoin sellers are always going to go on forever. Srikar Paruchuru · ▶ 16:00
Key Takeaways
- Two competing orchestrators sharing a deployment need a single arbitration object to stay consistent.
- Stale informer caches can flip a status to success early, advancing an irreversible state machine incorrectly.
- Integration tests against kind clusters caught race conditions that unit tests and fakes could not reproduce.
About the Speaker(s)
Srikar Paruchuru is a Senior Software Engineer at Uber building the next generation of Uber’s Compute Platform. When not debugging controller races or migrating control planes, you will find him travelling or lounging about with a nice cup of coffee. He describes himself as a Distributed Systems enthusiast for whom Kubernetes is the latest obsession.
Egor Grishechko is a Senior Software Engineer at Uber on the same stateless compute team. His self-described specialty: professionally causing outages.