AI
The GPU Observability Gap in Kubernetes
A single H100 costs nearly $4 an hour on a major cloud provider. Scale that to eight devices per node and the bill exceeds $30 an hour. Eight nodes for a month runs past $126,000, and that figure excludes traffic and storage. Despite that cost, the dominant observability tools, nvidia-smi and DCGM, export node-level hardware counters: utilization, memory consumption, temperature. There is no per-pod breakdown and no explanation of why a GPU sits idle.
Why Existing Tools Fall Short
Nvidia-smi gives hardware-level counters that are hard to correlate with application behavior. Nsight, Nvidia’s proprietary profiler, can inspect kernel execution timelines and set breakpoints, but it introduces 10 to 50% overhead and targets developers on workstations, not production clusters. Tools such as Polar Signals and Grafana Beyla use eBPF uprobes to collect kernel launch histograms and memory transfer rates, which helps. Even so, an operator watching thousands of traces across hundreds of pods still cannot quickly answer why a specific inference service is slow.
How eBPF Uprobes Hook Into the CUDA Driver
Processes communicate with the GPU through the device driver library, libcuda. A PyTorch training job that needs to launch a kernel calls cuLaunchKernel in that library, which dispatches work to the device. eBPF uprobes attach to the entry and exit points of those functions, capturing call frequency, arguments, and return codes for every process on a Kubernetes node. A profiling agent runs as a DaemonSet, detects when a container loads the Nvidia driver library, attaches the probes, and forwards events to a Prometheus exporter visualized in Grafana.
Three Metrics That Explain GPU Behavior
Raw CUDA traces tell you what happened. Three derived metrics tell you what it means. GPU demand measures kernel launches per second, the rate of productive work queued to the device. GPU back pressure measures time the host spends blocked waiting for the device in synchronization calls. GPU contention counts how many processes share the same device simultaneously. The diagnostic power sits in the relationship between these signals over time, not in any one figure alone.
Three Production Incidents Diagnosed With This Framework
Incident one: a PyTorch job slows after a “logging-only” deploy. Demand stays flat, back pressure rises, and device-to-host transfer climbs to 4 MB/s. The logging change pulled full tensor copies from GPU to host. Incident two: a real-time inference service sees P95 latency double. Demand and back pressure both drop while contention rises to two. A batch job landed on the same GPU. Incident three: a CPU limit adjustment halves training throughput. Demand collapses because the CPU bottleneck starves the data loader, leaving the GPU undersaturated.
eBPF Limitations and the Road Ahead
eBPF probes attach at the driver API boundary, not inside the device. That means kernel execution time cannot be reliably measured, per-process GPU utilization cannot be calculated cleanly, and bottlenecks inside kernel code stay invisible. Hardware vendors need to ship first-class observability APIs designed for low overhead, multi-tenancy, and cloud-native deployment. With those APIs in place, Kubernetes schedulers and optimization tools could place workloads more intelligently based on real device signals rather than coarse node-level counters.
Notable Quotes
a single H100 costs nearly $4 an hour on a major cloud provider Zahari Dichev · ▶ 02:23
the lightweight tools don’t really give you all that much information while that’s that’s useful while the heavyweight tools uh that are very very useful in diagnosing problems are not really meant to run in production at all. Zahari Dichev · ▶ 06:54
The diagnostic power is not in just individual signals. It’s the relationship between them and how how how these signals move over time. Zahari Dichev · ▶ 13:15
it’s all essentially kind of cut at the border of the device driver inter interface Zahari Dichev · ▶ 25:16
Key Takeaways
- An eight-node H100 cluster costs more than $126,000 per month, yet today’s tools offer no per-pod GPU visibility.
- Three eBPF-derived metrics, demand, back pressure, and contention, can distinguish application bugs from environment problems.
- Hardware vendors need low-overhead, multi-tenant GPU observability APIs built for Kubernetes before the tooling gap closes.
About the Speaker(s)
Zahari Dichev is a software engineer at Buoyant, the creator of Linkerd. He focuses on performance, distributed systems, and cloud-native technology.