Why the Five-Minute Wait Lasted Seven Years

▶ Watch (0:29)

CrashLoopBackoff was introduced in 2015. The exponential delay, capped at 300 seconds, had no deep benchmarking behind it. Original maintainers confirmed it was a quick defensive mechanism to protect the node, nothing more. By 2017, GitHub issue 57291 had accumulated over 360 positive reactions, making it the third most reacted issue in the core repository. Users wanted three distinct fixes: instant restarts for clean zero exits, faster recovery after short outages, and a manual reset for pods already stuck at five minutes.

How Teams Worked Around a Hardcoded Timer

▶ Watch (4:13)

Users hit the 300-second wall hard enough to bypass it at every layer of the stack. At the application layer, teams wrote bash wrapper scripts to hide crashes from Kubernetes entirely. At the operator layer, custom reaper operators polled the API to delete any pod entering backoff and force a fresh restart. At the infrastructure layer, some teams maintained private Kubernetes forks just to patch the one hardcoded constant and compile their own kubelet. Three different workarounds for one timer is a clear API gap.

Measuring the Physical Limits of the Kubelet

▶ Watch (10:19)

Before changing the defaults, Li’s team ran stress tests with 110 instantly crashing single-container pods on a saturated node. API traffic spiked only to the kubelet client-side rate limit of 5 QPS. Kubelet CPU usage doubled. The node survived. That data proved enforcing a 1-second minimum is safe. Restarting a container is a heavy lifecycle event: kubelet must kill the pod, unmount volumes, remove cgroups, then wait for network attachment, volume mounting, and image pulling before a new container starts.

The Node-Level Knob: KEP-5593

▶ Watch (11:23)

KEP-4603, introduced in 2024, aimed to change the global default backoff for all users. That required long alpha soak time. The community split the work. KEP-5593 extracted just the node-level setting and fast-tracked it. In Kubernetes 1.35 it reached beta. The new maxContainerRestartPeriod field sits in kubelet configuration, not in the pod spec. Putting the knob at the node level shifts control from individual developers to cluster operators. A dedicated ML node pool can now run at a 2-second maximum while standard web service nodes keep the 5-minute default.

Three Workload Demos, One Cluster

▶ Watch (15:05)

Li ran three scenarios across nodes configured at 2-second max backoff, 60-second max backoff, and the legacy 300-second default. In the task-isolation scenario, a queue worker exiting with code zero tore through tasks on the 2-second node while the 300-second node got trapped in backoff for doing its job too fast. In the transient-outage scenario, a web app reconnected to a restored database almost instantly on the 2-second node. In the sidecar scenario, a proxy container killed by OOM every 10 seconds kept the main app stuck in a network black hole on the legacy node.

Q&A

How does setting maxContainerRestartPeriod to 2 seconds produce a flat 2-second restart rather than a growing backoff? When the configured maximum is below the initial default of 10 seconds, the backoff never exceeds that cap, so it stays at 2 seconds on every restart. ▶ 24:07

Notable Quotes

There was no deep benchmarking. The current behavior was not designed beyond the motivation to throttle misbehaving containers. Yang Li · ▶ 02:21

when your users are doing all these things to just bypass a simple timer, you know, you have an API gap. Yang Li · ▶ 05:00

breaking a m massive ambitious cap into smaller practical pieces is the best way to solve a seven-year-old problem. Yang Li · ▶ 21:51

Key Takeaways

  • The 300-second CrashLoopBackoff cap was chosen in 2015 with no benchmarking behind it.
  • Stress tests on 110 crashing pods confirmed a 1-second minimum restart period is safe for node stability.
  • KEP-5593 reached beta in Kubernetes 1.35, giving operators a node-level maxContainerRestartPeriod field.

About the Speaker(s)

Yang Li is a Technical Solutions Engineer with Google Kubernetes Engine. He has been contributing to the Kubernetes open source project in his spare time since 2017, working across roles as an application developer, cluster administrator, and project contributor.