AI
A Decade of Workload Shifts, One Static Node Model
From 2016 to 2025, Kubernetes moved from web frontends to GPU orchestration. Features like in-place pod resizing and workload-specific node swap became standard. But the core node lifecycle model stayed static. TPU and GPU accelerators need precise hardware alignment. LLM training jobs run for weeks. Any idle time on a modern accelerator cluster costs money. Inference nodes carry terabytes of model weights that take 10 minutes just to warm up.
How Kubernetes Treats Planned Maintenance Like Node Failure
Lucy Sweet groups today’s node problems into signals, actions, and coordination. Kubernetes reports architecture (x86) and nothing else, blind to GPU driver versions and hardware generations. Two production clusters running identical hardware can use different label naming conventions, blocking cross-cloud portability. For maintenance, there is no first-class representation. Teams taint the node and drain it, treating planned work like an unexpected failure. There is no way to signal future intent: Kubernetes knows the current state, not what operators plan to do next.
Eviction Request: Observable Intent for Pod Removal
The original eviction API is binary: either the pod is removed or you get a 429. No observable state, no room for multi-step pre-eviction work. Eviction Request is the replacement. Creating the object targets a pod without removing it. Controllers watch for these requests, gate them, run their own logic (data migration, warming a replacement), then ungate. The pod then goes through standard eviction. The KEP is merged. Alpha targets Kubernetes 1.37.
From Label Chaos to Managed Node Capabilities
Node Declared Features replaces label fragmentation. Two clusters running the same GPU hardware often use different label naming conventions, blocking portable workloads and cross-cloud automation. The Node Status Declared Features API turns that chaos into managed intent, providing workload-scoped protection, portability, and zero-config scheduling. Node Readiness Gates ship as an out-of-tree component. Operators write readiness rules specifying what a node must satisfy before serving workloads. A node moves from idle to serving only once those rules pass.
Nodes as an Implementation Detail
GKE Autopilot and EKS Autopilot exist because users want applications to run, not to manage nodes. The working group’s direction moves Kubernetes abstractions away from node and pod toward higher-level workload primitives. The Workload API, introduced in v1alpha2 by sig-scheduling, schedules entire applications as a cohesive unit. DRA is being extended beyond individual devices to model node groups and infrastructure topology. When a node fails, a replacement with the same hardware signature can take its place automatically.
Q&A
What happens to nodes stuck in an unknown or undefined state? Dawn explained the root cause is usually a kubelet in a crash loop that cannot report heartbeats, or a network partition severing the control plane link. Improved network reliability and partition handling are on the roadmap. ▶ 28:15
How do stateful sets with volumes get handled during eviction? Lucy said the Eviction Request API could evolve to express intent around volumes, letting controllers move data before eviction begins rather than after. ▶ 29:18
Is the kubelet amnesia bug being addressed? Lucy confirmed the working group is investigating. Ryan Haly has a talk the same afternoon covering graceful node shutdown, which targets the pod rescheduling race condition directly. ▶ 30:48
Notable Quotes
Kubernetes should be boring Dawn Chen · ▶ 2:29
people are having to solve it themselves Lucy Sweet · ▶ 10:09
just want their applications to run. Lucy Sweet · ▶ 25:32
I’m not promising anything. Lucy Sweet · ▶ 30:02
Key Takeaways
- The taint-and-drain maintenance default blocks GPU nodes before hardware goes offline, wasting accelerator time.
- Eviction Request API adds observable, multi-step pod removal with controller gating, targeting alpha in Kubernetes 1.37.
- Node Declared Features replaces ad-hoc label conventions with managed intent, enabling portable scheduling across clusters.
About the Speakers
Dawn Chen is a principal software engineer at Google. She worked on Kubernetes and Google Container Engine (GKE) before the project was founded and has served as a tech lead in both Kubernetes and GKE.
Lucy Sweet is a Staff Engineer at Uber and co-lead of the Kubernetes Node Lifecycle Working Group. Her team manages one of the largest Kubernetes deployments in the world. She has helped shape upstream features including In-Place Pod Resizing and the Eviction Request API.