AI
The nvidia-smi Blind Spot That Crashes Training Jobs
nvidia-smi reports 6.71 GB free on the GPU. The training job crashes anyway, rejected while trying to allocate a 7.54 GB tensor. The monitor is not lying, but it is only reporting total free memory. It says nothing about whether that memory sits in one contiguous block. PyTorch needs one solid block for a large tensor. Scattered free pages do not qualify. That gap between what nvidia-smi shows and what the allocator actually sees is the root of the entire problem.
Why Contiguous Memory Is the Only Memory That Counts
Think of a school bus looking for a parking space. Five empty spots scattered across a lot are useless if the bus needs one long continuous space. GPU memory works the same way. Fragmentation leaves blue active-data blocks mixed with small free green fragments across the GPU. Sum those green fragments and you get 10 GB free, which nvidia-smi proudly reports. Ask PyTorch to allocate 7 GB in one shot, and it fails. The memory exists; the contiguous block does not.
Measuring Fragmentation With the Right Tool
The torch.cuda memory stats API tracks fragmentation directly. Subtract allocated memory from reserved memory. The remainder is the fragmentation ratio. Ifeanyi calls this the metric that actually reveals what is happening inside the GPU during training. nvidia-smi will never show it. Watching this number during a run surfaces fragmentation before it accumulates far enough to kill the job at 99%.
Reducing Memory Pressure Before Fragmentation Starts
Three techniques cut memory pressure at the code level. Shrinking batch size from 64 to 16 reduces peak allocations but costs throughput. Gradient checkpointing trades speed for lower memory usage during the forward pass. Mixed precision, either float16 or bfloat16, moves most GPU math from 32-bit to 16-bit, cutting the memory footprint for each operation roughly in half. Each option involves a tradeoff, so choosing depends on how badly fragmentation is hitting a specific training run.
Cluster-Level Defense With DaemonSets and Checkpointing
When code-level fixes are not enough, deploy a DaemonSet to pull fragmentation metrics and export them to Prometheus. Set a threshold and trigger an alert when that threshold is met. The next step is draining the affected node. Draining only works safely if the training code implements checkpointing, so the job resumes from where it paused rather than restarting from scratch. Without checkpointing, draining a node after hours of training wastes everything that ran before it.
Notable Quotes
the real story is that the Nvidia SMI basically is just reporting the um total free memory but it’s not telling you that you have the contiguous free memory. And that’s the distinction of the root of this entire problem. Michael Ifeanyi · ▶ 1:24
fragmentation has already started eating into the GPU. Michael Ifeanyi · ▶ 3:34
when you restart those nodes you want the job to continue from where it paused before you drain the node. Michael Ifeanyi · ▶ 5:39
Key Takeaways
- nvidia-smi reports total free memory, not contiguous free memory blocks.
- PyTorch tensor allocation fails when fragmentation breaks free memory into scattered chunks.
- Use
torch.cudamemory stats, not nvidia-smi, to track fragmentation ratio during training. - Batch size reduction, gradient checkpointing, and mixed precision each reduce peak memory demand.
- A DaemonSet exporting fragmentation metrics to Prometheus can trigger node drains before a job crashes.
About the Speaker(s)
Michael Ifeanyi is a Technical Solutions Engineer at Google specializing in Kubernetes, cloud security, and DevOps. He holds a Master’s in Cybersecurity and carries certifications including CISSP, AWS Solutions Architect, and GCP Professional Cloud Security Engineer.