AI
Why Checkpoint Restore Belongs in Kubernetes Now
GPU resources are expensive and scarce. Several companies have already built proprietary software stacks around checkpoint restore, focusing specifically on GPU checkpoint restore. That commercial activity signals the value. The working group’s case is that upstream Kubernetes needs native support so those benefits reach all users, not just teams who can build their own stack. The concept dates to roughly 2005 and has been in active Linux kernel development since around 2010.
Three Use Cases Driving the Design
The working group is targeting three concrete scenarios. First, accelerating application startup: save an initialized state once, skip re-initialization on every restart. This already shows up in AI inference serving and Java applications. Second, fault tolerance: periodic checkpoints for long-running HPC and stateful workloads so a node failure does not discard hours of computation. Third, resource utilization: checkpoint-based preemption instead of kill-based preemption, plus reclaiming GPU memory from idle workloads by moving state to host memory before loading a different job.
From Container Checkpointing to Pod Checkpointing
Container-level checkpointing shipped as beta enabled-by-default in Kubernetes 1.30. The path there started with Adrian Reber opening a pull request, getting pushed back to write a KEP first, and then narrowing scope to forensic container checkpointing. OCI checkpoint images are now built consistently across CRI-O, containerd, and Podman using the same annotation pattern. Pod-level checkpointing is the next step because the pod is the smallest deployable unit in Kubernetes. The working group has two proof-of-concept implementations: one modifying the controller manager in-tree, one using an external snapshot controller.
API Design Choices and Security Trade-offs
The proposed flow has a user creating a PodCheckpoint object as a request, with a controller accepting it and executing the checkpoint. Restore is a separate object, not a field on the pod spec. The SIG API Machinery meeting pushed back on embedding restore in the pod spec: if a checkpoint is deleted after restore, the reference breaks. There is also a permission gap. A user might have rights to create a pod but should not automatically be allowed to restore from a checkpoint, since the checkpoint archive exposes the entire memory space of the pod including secrets.
What the Working Group Has Left Open
The KEP did not merge in Kubernetes 1.36. The group is targeting concrete progress in 1.37. Open questions span multiple SIGs. SIG Node asks how to cover both CRIU and gVisor without an overly generic API. SIG Auth asks who may checkpoint or restore a pod and whether namespace-scoped policy should differ from cluster-scoped policy. Storage questions include where checkpoint archives land when checkpointing 10,000 pods or GPU memory, and whether the controller or the kubelet moves checkpoint files between nodes. IP address retention for suspend-resume remains unsolved.
Q&A
What proprietary implementations already exist in production? Several companies use container-d to implement checkpoint restore and extend Kubernetes runtimes; IP address retention after migration and GPU driver version mismatches are the most common challenges they report. ▶ 27:09
Could checkpoint restore support crash-consistent storage backups by freezing pods? The pod is frozen during checkpointing, and the group has discussed adding an explicit paused pod state to Kubernetes, similar to what Docker and Podman already support. ▶ 29:54
Can a user checkpoint only the warm-up stage of a Python batch job rather than the full pod? Serverless platforms already do this in production; the initial KEP will checkpoint the entire pod, but partial checkpoints are noted as relevant for later work. ▶ 31:39
How does checkpoint restore relate to application-consistent backup strategies? The boundary is not clearly defined; distributed checkpoints require coordinating freezes across multiple pods simultaneously, and the working group meets every Thursday to work through these questions. ▶ 33:44
Notable Quotes
inefficient CPU and GPU utilization is a persistent issue of cloud infrastructures and checkpoint restore could be used as a primitive for load balancing between nodes or for non-disruptive preeemptions of lower priority workloads Viktória Spišaková · ▶ 06:22
if you checkpoint a pod and you’re able to inspect the checkpoint archive you can see the entirety of the memory space. You can see all of the you know all the basically everything that the pod was up to. Peter Hunt · ▶ 22:28
we created the working group so that we can combine the input from different people and have an open discussion about it Radostin Stoyanov · ▶ 29:39
in 136 we didn’t we didn’t even get uh enhancement merged. So we’re still very much in the beginning stages pre-alpha of this feature. Peter Hunt · ▶ 20:50
Key Takeaways
- Container-level checkpointing is beta-enabled-by-default in Kubernetes 1.30, using consistent OCI image annotations across CRI-O, containerd, and Podman.
- Pod-level checkpointing uses a separate PodCheckpoint request object; restore is a distinct object to enforce security permission boundaries.
- The KEP missed Kubernetes 1.36 and targets 1.37, with IP migration, storage backends, and multi-SIG API design still unresolved.
About the Speaker(s)
Radostin Stoyanov is a PhD student in the Scientific Computing research group at the University of Oxford. His research focuses on improving the resilience and performance of HPC and cloud computing systems.
Peter Hunt is a Principal Software Engineer at Red Hat. He chairs SIG Node and the WG Checkpoint Restore, and maintains CRI-O.
Viktória Spišaková is a PhD student at FI MUNI and an IT architect at ICS MUNI, with several years of experience running Kubernetes for e-INFRA CZ.
Adrian Reber is a Senior Principal Software Engineer at Red Hat. He has been migrating processes since 2010, starting in high-performance computing, and earned a PhD for that work.