StatefulSet MaxUnavailable: Shipped, Then Disabled

▶ Watch (1:18)

MaxUnavailable for StatefulSets reached beta in 1.35, enabled by default, then got disabled a week before KubeCon. A user posted on Slack describing failures they observed. That report let Maciej and Phillip trace the exact bug. The fix changes about 20 lines of code but required nearly 200 lines of new tests. Kubernetes 1.36 ships with the feature off. The patch point release of 1.35 also disables it. Users relying on the feature must enable the feature gate manually until 1.37 or 1.38.

Terminating Pods in Deployments: A Two-Year Problem Splits in Two

▶ Watch (3:44)

Deployment pod replacement policy has been under discussion for roughly two years. The team separated two concerns to move faster. A new status field for terminating replicas in Deployments and ReplicaSets graduates to beta now. The broader pod replacement policy, which changes how both the ReplicaSet and Deployment controllers operate, stays in active discussion. Phillip, who implemented both pieces and was present at the talk, received credit for the work. The split lets one feature ship while the riskier controller changes get more testing time.

Job Improvements for Long-Running and Large-Scale Workloads

▶ Watch (7:21)

The Batch Working Group, a SIG Apps spinoff focused on ML and HPC workloads, drove several Job changes. Users can now suspend a Job, wait for all running pods to terminate, then modify CPU, GPU, and memory resources before resuming. This targets training jobs that run for a week or more. Backoff limit per index lets a single failed index exhaust its own retry budget without failing the whole job. Combined with the job success and completion policy, a job with 100,000 pods can treat one failed index as ignorable if 99% of executions succeed.

External Job Management and the ManagedBy Field

▶ Watch (13:09)

Kueue, a queuing and fair-sharing project maintained by Miha and Kevin, needed to replicate Jobs across worker clusters while keeping a summary Job in a central hub cluster. The default Job controller would interfere with that model. The solution is a spec.managedBy field that tells kube-controller-manager to leave the Job alone. What looks like a simple annotation required auditing the entire Job status specification and adding API-server validation to enforce consistent status reporting across any external controller. Inconsistencies in the default controller itself surfaced and were fixed in the process.

Agent Sandbox and MCP Lifecycle Operator: New SIG Apps Sub-Projects

▶ Watch (20:25)

Agent Sandbox, announced at KubeCon NA, is a controller that creates one pod per sandbox with an attached service. A warm pool of pre-created pods cuts bootstrapping time when a sandbox is claimed. Janet demoed exec-ing into a running sandbox to interact with OpenClaw, keeping agent workloads off the local machine. A second sub-project, MCP Lifecycle Operator, manages the lifecycle of MCP servers inside a cluster. It started about one month before this talk, with a first release expected within days. Both projects target AI agent runtime use cases on Kubernetes.

Q&A

Does inference pool fall under SIG Apps or a different group? A dedicated Serving Working Group handles inference use cases and works closely with SIG Network, touching SIG Apps only when workload API changes are needed. ▶ 26:53

What is the use case for the new StatefulSet recreate update strategy? StatefulSets originally required manual intervention on any stuck rollout to protect data, but many users adopted them only for stable pod naming, so recreate destroys the full set and restarts pods under the existing ordered or parallel policy. ▶ 28:02

Notable Quotes

the code changes is like 20 lines of code but I’m pretty sure we are close to 200 lines of just tests Maciej Szulik · ▶ 06:14

it was actually uh some person showing up on our slack like listen this is the problem that I were seeing in a max and available in a stateful set Maciej Szulik · ▶ 03:02

I can do anything I want without worrying about this is going to you know put a security risk on my own machine. Janet Kuo · ▶ 24:16

it looks simple but it wasn’t. Maciej Szulik · ▶ 18:23

Key Takeaways

  • MaxUnavailable for StatefulSets is disabled in 1.35 and 1.36 pending edge-case fixes targeted at 1.37.
  • Jobs now support suspend-then-modify for CPU, GPU, and memory, serving week-long training workloads.
  • The new spec.managedBy field lets external controllers like Kueue manage Jobs without interference from kube-controller-manager.
  • Agent Sandbox and MCP Lifecycle Operator are active SIG Apps sub-projects for running AI agent workloads on Kubernetes.

About the Speaker(s)

Janet Kuo is a Staff Software Engineer at Google Cloud and has contributed to Kubernetes since before its 1.0 launch in 2015. She serves as a Kubernetes maintainer and co-leads SIG Apps as both chair and tech lead.

Maciej Szulik is a Staff Platform Engineer at Defense Unicorns with more than 20 years of software experience. He co-leads SIG Apps alongside Janet and works on Kubernetes challenges by day while tinkering with Python projects outside of work.