AI
Why the Existing Eviction API Falls Short
Pod termination today arrives from many sources at once: the scheduler, kubelet, drain controllers, and direct deletes. PDBs help for high-availability applications, but a single-replica database or virtual machine has no way to express a wish for graceful termination. Operators work around this by setting PDB min-unavailable to zero or writing admission webhooks. Both approaches work but give no visibility into whether the eviction will complete or how far along it is.
Eviction Request API: Declarative Eviction with Custom Responders
The Eviction Request API converts the imperative eviction call into a declarative object. The object stores the pod UID (which may change during a long eviction window) and tracks all requesters. Custom “responders” register against the pod and run before any termination fires. A responder might create a new replica set, drain a database, or wait for a scale-out. In alpha one, each responder must send a heartbeat at least every 20 minutes or the controller moves to the next responder. The fallback is always the original imperative eviction, so existing behavior is preserved. The feature missed the 1.36 code freeze and is targeted for 1.37.
Live Demo: PDB Blocking, Migration Responder, and Cancellation
Filip showed three flows against a live cluster. First, a default eviction with no responders terminated a pod immediately. Second, a PDB with min-available one blocked the eviction, which entered exponential backoff and reported the budget violation in the status. Third, a custom responder created a new replica set called “DB migrated,” waited for it to become available, then let the original pod terminate. A fourth demo showed an administrator withdrawing their eviction intent mid-flight, producing a failed eviction request because no requester remained. Server-side apply is recommended for status updates because multiple actors write concurrently.
Node Maintenance Observability: The Fragmented Drain Problem
More than a handful of tools implement node drain today, each with its own opinion about what “drained” means. When kubectl drain runs on a laptop, cluster components cannot tell whether a node is being drained or whether pods are simply moving for another reason. State stays on the client side. The Specialized Lifecycle Management KEP addresses this in three steps: first, a shared place to publish node state; second, agreed-upon state names with a rough common definition; third, adoption by core controllers like cluster-autoscaler and external tools like node-problem-detector.
Kubelet Amnesia During Graceful Node Shutdown
During graceful node shutdown, the kubelet drains pods and then may be interrupted before finishing. When that happens, it restarts as a normal kubelet and begins accepting pods again, losing track of the in-progress shutdown. Ryan Hallisey called this “kubelet amnesia.” The fix requires the kubelet to observe an external record of its intended state so it can resume draining after a restart. Specialized Lifecycle Management is the proposed home for that state. The working group is keeping scope narrow to avoid spreading effort across too many features simultaneously.
Q&A
Will kubectl drain and cloud node-upgrade operators adopt the Eviction Request API automatically when 1.37 ships? Preliminary discussions with integrators are positive, they want the feature because it offloads eviction-alignment bugs to the API layer, but kubectl drain will not use it on day one. ▶ 31:27
Notable Quotes
this could have been drain out an application’s data if it’s like a database. This could have been make sure there is a new replica available before deleting the old one. This could have been any business logic that we want to program. Lucy Sweet · ▶ 11:44
unless your application doesn’t respond in 20 minutes, you are given as much time as you want to basically migrate. Filip Křepinský · ▶ 07:39
the kubelet goes back to being a normal kubelet. It’s it can start taking pods in again, right? And and so, it actually has state, right? There’s state. It’s it’s a graceful node shutdown kubelet state, you know, whatever you want to call it. It should be doing graceful node shutdown, and it forgets. Ryan Hallisey · ▶ 28:20
it? 400 people in this room? And uh when I go to the Zoom meetings, there’s eight, so I feel like at least some of you should be turning up. Lucy Sweet · ▶ 29:55
Key Takeaways
- Eviction Request API targets Kubernetes 1.37 as a declarative layer above PDBs.
- Custom responders can run migrations lasting hours before any pod termination fires.
- Kubelet amnesia causes graceful-shutdown state loss on restart; Specialized Lifecycle Management KEP aims to fix this.
About the Speaker(s)
Filip Křepinský is a Principal Software Engineer at Red Hat working on the OpenShift project. He led the Eviction Request API design and live demos at this talk.
Lucy Sweet is a Staff Engineer at Uber and co-lead of the Kubernetes Node Lifecycle Working Group. Her team manages one of the largest Kubernetes deployments in the world. She has helped shape upstream features including In-Place Pod Resizing, the Eviction Request API, and Pod Level resources.
Ryan Hallisey is a Principal Software Engineer at NVIDIA, building data centers powered by Kubernetes and KubeVirt for NVIDIA products. He presented the node maintenance observability approach and the kubelet amnesia problem.