AI
Why Distributed Training Failures Cost Millions
A protein structure prediction job at Czech National Research Infrastructure ran for 34 days on a single GPU before the node overheated and the job died. No checkpoint, no recovery. All 34 days of compute lost. Scaled to 1,000 GPUs, that failure mode becomes catastrophic. One study found 460 interruptions during 54 days of training, with nearly 80% caused by hardware errors. Another measured mean time between failures at three to 23 hours on older GPUs. Together those failures add up to millions of dollars in lost compute.
The Gang Scheduling Problem
Distributed training jobs require gang scheduling: either every task starts or none can. Gang scheduling is expensive. Once a gang is running, any single task failure can stall or kill the whole job. This makes a distributed job exactly as reliable as its weakest infrastructure component. Failure categories include infrastructure errors (network timeouts, InfiniBand errors), AI engine errors (CUDA initialization failures, out-of-memory), and user errors (misconfigurations, permission errors). Infrastructure and user errors tend to appear early. Hardware errors surface later and dominate unexpected interruptions.
GPU Checkpointing: How It Works and Where It Slows Down
The GPU driver exposes four operations to user space: lock computation, move GPU state to host memory, restore from host memory back to GPU, and unlock. Checkpointing a training job saves a full memory dump of the application. Writing all that GPU state to disk is the bottleneck. The team integrated compression directly into the checkpoint pipeline, compressing memory pages before writing and skipping pages that contain only zero bytes. For inference workloads this reduces checkpoint size significantly and cuts restore time by reading less data from disk. For training jobs, compression yields a 10 to 20% reduction in checkpoint size.
Deduplication: 90% Smaller Checkpoints for Quantized Fine-Tuning
H100 and B200 GPUs carry large memory states. CPU workloads use a soft dirty bit in the page table to track which memory changed since the last checkpoint. That does not work for GPU state because checkpointing moves GPU memory into new host allocations each time, making change tracking impossible. The team’s solution computes checksums for GPU memory pages, stores them with the checkpoint, and on the next iteration skips pages whose checksums match. For quantized fine-tuning the frozen model weights never change. Skipping them reduces checkpoint size by 90%. The benefit is smaller for full-precision training where most weights update every iteration.
Kubeflow TrainJob Integration and the Suspend-Resume Demo
The team extended the Kubeflow TrainJob controller to trigger checkpointing whenever the suspend flag is set to true. The demo trains a GPT-2 model across two communicating pods using PyTorch and Kubeflow Trainer. Triggering suspend causes the controller to checkpoint each pod, save archives to local disk, and convert them to OCI images. On resume, the controller restores from those images using the same IP addresses assigned during the original run. NCCL communications must be fully aborted before the checkpoint and rebuilt after restore. Low-priority jobs preempted by the scheduler also hit this path automatically.
Q&A
How mature is this and is it available in Kubeflow Trainer today? The work is research-stage; patches exist but still need community review, approval, and integration into Kubeflow Trainer and Kubernetes job sets. ▶ 18:41
Notable Quotes
After 34 days of running, the node just overheated, the job got stuck and reboot was necessary. This means that the job um was just lost. Viktória Spišáková · ▶ 00:44
a distributed job is only as reliable as a least reliable infrastructure component. Viktória Spišáková · ▶ 02:31
we can reduce um checkpoints by 90% the size of checkpoints by 90% by using this dead duplication mechanism. Radostin Stoyanov · ▶ 12:30
essentially it’s a long process Radostin Stoyanov · ▶ 20:32
Key Takeaways
- A single node failure after 34 days of GPU compute motivates infrastructure-level checkpointing without user code changes.
- Checksum-based deduplication cuts checkpoint size by 90% for quantized fine-tuning where frozen weights never change.
- Kubeflow TrainJob suspend-resume now triggers transparent GPU checkpoints, handling NCCL teardown and OCI image conversion automatically.
About the Speaker(s)
Andrey Velichkevich is a Senior Software Engineer at Apple and a key contributor to the Kubeflow project. He sits on the Kubeflow Steering Committee and co-chairs the Kubeflow AutoML and Training WG, and is an active member of the CNCF WG AI.
Viktória Spišáková is a PhD student at FI MUNI and IT architect at ICS MUNI, with several years of experience providing Kubernetes infrastructure at e-INFRA CZ.
Radostin Stoyanov is a PhD student in the Scientific Computing research group at the University of Oxford. His research focuses on improving the resilience and performance of HPC and cloud computing systems.