AI
The Day-Two Gap in Kubernetes Stateful Workloads
Kubernetes handles day-one operations for stateful workloads well. Persistent volume claims, persistent volumes, StatefulSets, and Deployments all exist. Day-two operations, meaning backup and recovery, are still limited. The working group formed specifically to fill that gap. A backup requires two things: Kubernetes metadata and the data inside persistent volumes. Restoring means importing that metadata, re-creating PVCs and PVs, and either replaying a native data dump or rehydrating from a volume snapshot.
COSI: Object Storage Buckets as First-Class Kubernetes Resources
COSI, the Container Object Storage Interface, lets a user provision an object bucket the same way a PVC is provisioned. It is moving to v1 alpha 2 in Kubernetes 1.26. The v1 alpha 2 release adds multi-bucket-claim access, so a single BucketAccess object can now reference more than one BucketClaim. It also introduces three access modes, read-write, read-only, and write-only, across three data types: object data, object metadata, and bucket metadata. Admins can allow or disallow specific combinations inside a BucketAccessClass.
Volume Group Snapshots: Consistent Point-in-Time Copies Across Multiple Volumes
Some applications write across multiple volumes and need all of them snapshotted at the same instant for write-order consistency. Volume group snapshots, tagged GA in Kubernetes 1.26, solve this. An admin labels the target PVCs, then a VolumeGroupSnapshot object selects them by label. A new group controller service and new gRPC interfaces in the CSI spec handle the actual snapshot call. Each individual volume snapshot produced by the group snapshot can be used directly to create a new PVC.
Change Block Tracking: Efficient Incremental Backups Without Vendor-Specific APIs
Without change block tracking, a backup vendor must call each storage vendor’s proprietary API to find what changed between two snapshots. That is inefficient and couples the backup tool to every storage backend. The working group added vendor-agnostic Kubernetes APIs, a new snapshot metadata service, and new CSI RPCs that identify changed blocks between two snapshots of the same persistent volume. This feature is targeting beta in Kubernetes 1.36 and lets backup vendors decouple their systems from the underlying storage.
Operator Backup Challenges: Resource Ordering, Quiescing, and Nested Operators
Operators reconcile top-down. If a restore job creates a database custom resource before the underlying PVC is in place, the operator creates a fresh empty database instead of the expected one. The fix is bottom-up restore order: PV first, then PVC, then StatefulSet, then the custom resource. Quiescing the operator during backup or restore removes the race condition entirely, but quiesce operations are not standardized today. The Harbor operator example illustrates a harder case: Harbor calls the Postgres and Redis operators, which means backup tooling must resolve the full dependency hierarchy before any recovery can start.
White Paper and How to Get Involved
The working group has published one white paper covering why Kubernetes backup is needed. A second paper is in progress and will prescribe best practices for admins, developers, and operator authors, including standard ways to expose quiesce operations and manage backup lifecycle across nested operators. The group meets every two weeks on Wednesdays. Application developers, cluster admins, and Kubernetes internals contributors are all welcome. Input on the second white paper is actively requested.
Notable Quotes
day two operations for stateful workloads such as data protection are still limited. Xing Yang · ▶ 01:06
That winds up with a race condition with the restore operation. So, you can wind up with a fresh empty database instead of a database containing the data you expected. Dave Smith-Uchida · ▶ 13:57
some operators have quiesce operations. Uh it’s not standardized, so you wind up writing custom scripts that have to be executed. Dave Smith-Uchida · ▶ 15:59
we need to understand this entire hierarchy. We need to understand who is responsible for doing the backup of what parts, and then we need to be able to orchestrate the recovery of this. Dave Smith-Uchida · ▶ 18:17
Key Takeaways
- Volume group snapshots reached GA in Kubernetes 1.26, covering multi-volume write-order consistency.
- Change block tracking targets beta in Kubernetes 1.36, replacing per-vendor storage APIs with a single CSI interface.
- Operators without standardized quiesce hooks and owner references create race conditions during backup and restore.
About the Speaker(s)
Dave Smith-Uchida is a Technical Leader at Veeam, where he works on the Kasten product for Kubernetes backup and recovery. He is a founding member of the Kubernetes Data Protection Working Group.