AI
Why Individual PVC Snapshots Corrupt Multi-Volume Databases
A PostgreSQL, MongoDB, or Cassandra deployment typically spreads data across at least three PVCs: a data volume, a write-ahead log volume, and a config volume. When Velero snapshots each one individually, the data PVC is captured at t=0, the WAL PVC at t=1, and the config PVC at t=2. Ten writes can happen between t=0 and t=1. The data volume knows nothing about those writes. The WAL does. Restoring that backup produces a database state that never existed, and corruption follows. The backup reported success the whole time.
The VolumeGroupSnapshot API: One Call, Atomic Capture
VolumeGroupSnapshot was introduced via KEP-3476, promoted to v1beta1 in Kubernetes 1.32, beta2 in 1.34, and is slated for GA in 1.36. The API adds three objects mirroring the existing snapshot hierarchy: VolumeGroupSnapshotClass, VolumeGroupSnapshot, and VolumeGroupSnapshotContent. A user selects volumes via label selector. The CSI sidecar issues one CreateVolumeGroupSnapshot call to the driver. The driver snapshots every volume in the group atomically and returns individual snapshot IDs. Crash consistency means all volumes are captured at the same instant, equivalent to pulling the power plug, with recovery handled by the app’s own WAL or journal replay.
Velero’s Implementation: Group Snapshots as Transient Orchestration
Velero 1.17 added VolumeGroupSnapshot support as opt-in. Add one label, velero.io/volume-group: db1, to each PVC that belongs to a group. Unlabeled PVCs take the existing individual snapshot path unchanged. Velero groups labeled PVCs into an item block, issues one CSI call, waits for individual VolumeSnapshots to become ready, then deletes the VolumeGroupSnapshot and VolumeGroupSnapshotContent on purpose. Restore targets need only standard VolumeSnapshot CRDs, not VolumeGroupSnapshot CRDs. The CLI command does not change: velero backup create. No new flags.
War Story: Ceph RBD Broke Restore with a Stale Handle Field
Testing against a Ceph RBD cluster produced a restore error after backup succeeded cleanly. Ceph’s RBD driver populates a volumeGroupSnapshotHandle field on every VolumeSnapshotContent created via a group snapshot. Velero had deleted the VolumeGroupSnapshotContent, so the snapshot controller could not find the matching object and failed reconciliation. The fix: Velero’s CSI plugin captures the handle as an annotation at backup time. On restore, a new restore action detects the annotation and creates a stub VolumeGroupSnapshotContent with matching handles. Once all VSCs reach ready state, the stub is deleted. Ceph RBD restore now works.
Benchmark Results: 266 Seconds of Inconsistency vs. Zero
Tests ran on Kubernetes 1.32 with CephFS, 1 GiB PVCs, and Velero 1.17. At 3 PVCs, VolumeGroupSnapshot was 1.3x faster than individual snapshots. At 10 PVCs, 1.5x. At 20 PVCs, 1.6x. At 50 PVCs, individual snapshots took 266 seconds total. That is 4 minutes and 26 seconds during which writes land between snapshots, each one widening the inconsistency window. VolumeGroupSnapshot issued one CSI call and captured all 50 volumes at the same instant. The demo backup of three PVCs completed in 14 seconds.
Adoption Checklist and What Comes Next
Five prerequisites, mostly one-time setup: Kubernetes 1.32 or later, VolumeGroupSnapshot CRDs installed, a CSI driver that supports the feature, a default VolumeSnapshotClass (a common gotcha the snapshot controller requires), and Velero 1.17 or later. The feature took roughly one year from a design doc in March 2025 through PVC grouping, core workflow, volume policy integration, and the Ceph RBD restore fix in February 2026. The VolumeGroupSnapshot API targets GA in Kubernetes 1.36. More CSI drivers are being validated; each new driver uncovers new edge cases.
Notable Quotes
your backup now contains a state that never existed in your application in reality right ever Shubham Dilip Pampattiwar · ▶ 03:54
your backup reported success everything looked green right it’s your restore that fails so individual snapshots in such scenarios give you a false sense of security Shubham Dilip Pampattiwar · ▶ 04:44
we use volume group snapshots as an orchestration mechanism in the backup workflow, not as a new backup format. Shubham Dilip Pampattiwar · ▶ 09:10
drivers are not uniform. There could be subtle or not so subtle differences in behavior from one driver to another. The volume group snapshot API defines what to do. However, each driver decides how to do it. Scott Seago · ▶ 16:12
the performance win is just a bonus. I think the consistency win is where you you should use volume group snapshots for Shubham Dilip Pampattiwar · ▶ 18:51
Key Takeaways
- At 50 PVCs, individual snapshots leave a 266-second inconsistency window; VolumeGroupSnapshot leaves zero.
- Velero deletes VolumeGroupSnapshot objects after backup so restore targets need only standard VolumeSnapshot CRDs.
- Ceph RBD requires stub VolumeGroupSnapshotContent objects at restore time; test each CSI driver separately.
About the Speaker(s)
Shubham Dilip Pampattiwar is a Principal Software Engineer at Red Hat, building open-source solutions for Kubernetes data protection and disaster recovery. He maintains the Velero project and contributes to backup and snapshot orchestration for Kubernetes clusters.
Scott Seago is a Principal Software Engineer at Red Hat and a maintainer of the Velero project. He is also an upstream contributor to the OADP project and a member of the OpenShift development team at Red Hat.