AI
From kubectl drain to 1,000 Drains per Day
Three years ago NVIDIA drained nodes manually and almost never. Today the team executes roughly 1,000 node drains every single day across a GPU data center running a latency-sensitive game-streaming workload. Device failures, driver failures, and both planned and unplanned maintenances arrive constantly. Each drain carries an operational cost measured in GPU hours. The goal driving every design decision is minimizing that cost while keeping the fleet transparent to end users who should never notice maintenance is happening.
Why kubectl drain Is Only the Beginning
kubectl drain cordons a node, evicts workloads, and stops. That covers three steps. NVIDIA identified many more states worth managing: scheduling when to drain, ranking which node to drain first, monitoring cordon status, performing the actual maintenance, and returning the node to service. Failing to own those states causes silent failures. When a drain fails on a laptop, only the engineer running the command knows. At scale that means Slack messages, not automation. Building a full lifecycle forced the team to go well beyond the base tool.
Two Controllers, One CRD: The System Architecture
Two microservices divide responsibility cleanly. The drain coordinator, called Drainator, operates at cluster scope. It receives maintenance requests, creates a CRD per target node, picks which nodes to drain using a ranking pipeline, and monitors drain completion. The lifecycle orchestrator operates at single-node scope. It drives state transitions, enforces SLAs, and sends notifications. The two controllers never communicate directly. All coordination flows through the maintenance CRD, which mirrors node annotations, labels, and cordon status into one queryable API.
Smarter Node Selection: Ranking and Diversity Injection
Drainator scores every candidate node with a composable ranking pipeline. Already-drained nodes earn points. Urgent unplanned maintenances earn more. Nodes with fewer active sessions earn more still. Without diversity injection the system would drain idle nodes first, finish quickly, then face a wave of loaded nodes all starting drain simultaneously. The fix mixes a small percentage of active nodes into each batch. By the time idle nodes finish maintenance, the active nodes have already begun draining and are ready sooner for the next iteration.
Self-Tuning Throttle and Stuck-Drain Recovery
The throttle has no fixed ceiling. Each successful maintenance opens it slightly. Each failure tightens it. Under a cascade of failures the system slows to a crawl automatically, giving engineers time to investigate before every node lands in a failed state. Stuck drains get two independent timeouts: one for drain complete but no maintenance process started, another for maintenance started but never finished. In both cases the system uncordons the node, requeues the maintenance, and writes timestamps to the CRD so there is an audit trail without human intervention.
Safety Guards and the Node Lifecycle Working Group
Several guardrails prevent compounding failures. Controllers only uncordon nodes they cordoned themselves, preventing accidental interference. Admission webhooks block duplicate maintenances and reject requests with empty node selectors, making it impossible to accidentally target an entire cluster. Metrics track phase transitions, completion rates, and failure counts. Ryan also leads the Kubernetes Node Lifecycle Working Group alongside contributors from Uber and Red Hat. Two KEPs, including the eviction request API merged for 1.36, aim to bring these patterns into upstream Kubernetes.
Notable Quotes
in a single day we will do about a thousand node drains. Ryan Hallisey · ▶ 00:19
wants to send the request and forget about it. not to babysit that request, not to check on did the node um drain what is going on, how the maintenance is going. Natalie Bandel · ▶ 11:44
if everything starts failing, we really fast we get into very slow process and it helps uh people uh check what is going on. Natalie Bandel · ▶ 21:52
treat maintenance as a first class life cycle. Ryan Hallisey · ▶ 28:06
Key Takeaways
- Separate the drain coordinator from the per-node lifecycle orchestrator to limit blast radius.
- A self-tuning throttle prevents maintenance work from triggering a production incident.
- Stuck-drain timeouts with auto-remediation eliminate the silent failures that kubectl drain leaves unresolved.
About the Speakers
Ryan Hallisey is a Principal Software Engineer at NVIDIA. He works on building data centers powered by Kubernetes and KubeVirt for NVIDIA products.
Natalie Bandel is a Senior Software Engineer at NVIDIA. She works on building software for cloud infrastructure powered by Kubernetes, KubeVirt and strong coffee.