AI
Why Immutability Has Limits in Real Clusters
Cluster API treats machines like immutable pods: every change is a create followed by a delete. That model prevents configuration drift and keeps machines aligned to a trusted image, which simplifies troubleshooting. But immutability creates friction. VMs take time to provision, GPU pools run dry, single-replica applications go down during drains, and long-running jobs must finish before a node can be replaced. These pressures motivated the team to look beyond pure create-and-delete rollouts.
How Disruption Budgets Drive the In-Place Decision
Before attempting an in-place update, Cluster API checks two spec fields: max surge (additional machines allowed during a rollout) and max unavailable (how much disruption is permitted). If the current availability count leaves no buffer, Cluster API scales up first to create room, then considers in-place. If the buffer exists, it contacts the update extension and asks whether it can handle every pending change. A partial answer forces a fallback to a full immutable rollout. This keeps the system safe without requiring user intervention.
The Update Extension Contract
The update extension is a new pluggable component. Cluster API sends it the full diff of desired changes and waits for a yes or no on whether it can handle all of them. If a user changes both an SSH key and a kernel version in one apply, and the extension only covers SSH keys, Cluster API falls back to rolling the machine. The core project has no opinion on how the extension performs its work. SSH commands, privileged pods, machine agents, and A/B deployments all satisfy the contract.
In-Place Rollouts for Control Plane and Worker Machines
Control plane machines only move one at a time. With max surge one and three replicas, Cluster API creates a fourth machine to build a buffer, then in-place updates two of the original three and deletes the extra. With max unavailable one, it skips the scale-up and updates all three sequentially. Worker machines follow machine deployment semantics: a machine moves from the old machine set to the new one before the in-place update starts. With max unavailable one and three replicas, no scale-up is needed and all three machines update in sequence without creating any additional infrastructure.
Chained and Efficient Upgrades Cut Worker Rollout Steps by 66%
Also shipped in the December release: chained upgrades let a user bump the cluster version by multiple Kubernetes minor versions in a single spec change. Cluster API walks through each intermediate control plane upgrade automatically. The companion feature, called efficient upgrades, skips intermediate worker machine rollouts entirely. Workers only upgrade when the control plane reaches the n-minus-3 version skew limit set by Kubernetes policy. Moving from 1.31 to 1.34 in one step means worker machines upgrade once instead of three times, reducing worker upgrade work by 66%.
When to Use In-Place Updates and When to Avoid Them
In-place updates suit changes that do not require pod drain or restart, such as rotating an SSH key. Kernel updates still require a drain, and at that point creating a fresh machine is better. A new machine starts from a known image, eliminates drift, and resets the security surface. The team’s stronger warning: do not use in-place updates to paper over broken drain support or slow VM provisioning. Drain is a foundational Kubernetes primitive, and avoiding it means fighting the platform.
Notable Quotes
you should not use in place up update as an excuse to avoid solving foundational problem on your system Fabrizio Pandini · ▶ 26:44
If you try to avoid drain is sort that you is sort of like that you are trying to fight Kubernetes Fabrizio Pandini · ▶ 27:08
update extension is a new component in the cluster API ecosystem and we assume that it will start small Fabrizio Pandini · ▶ 10:50
66% less work upgrades Stefan Büringer · ▶ 25:30
Key Takeaways
- Cluster API now chooses between immutable rollouts and in-place updates based on max unavailable and max surge budgets.
- The update extension contract lets providers implement in-place support incrementally without changes to the core project.
- Efficient upgrades reduce worker machine upgrade steps by 66% when jumping multiple Kubernetes minor versions.
About the Speaker(s)
Fabrizio Pandini is a Principal Engineer at Broadcom and SIG Cluster Lifecycle tech lead. He focuses on making Kubernetes lifecycle consistent across infrastructure types and maintains the Cluster API project.
Stefan Büringer is a Principal Engineer at Broadcom. He has contributed to Kubernetes since 2019 and maintains both Cluster API and controller-runtime.