AI
Why DCGM Metrics Leave Root Causes Hidden
NVIDIA DCGM reports GPU utilization, power draw, temperature, and error counts. Those signals come from hardware only. When two instances run the same model and one sits at 80% utilization while the other is stuck at 20%, DCGM cannot explain the difference. The tool shows what happened, not why. Problems that originate in the CUDA runtime or driver layers, between the application and the GPU hardware, stay invisible to hardware-level monitoring.
The Security Blind Spot at the CUDA Runtime Layer
The same visibility gap creates a security risk. Without insight into CUDA runtime calls, unauthorized kernel execution inside a container goes undetected. A container could run crypto mining workloads on GPU capacity that was allocated for inference. Abnormal memory transfers, detectable as spikes in cudaMemcpy traffic, can signal data exfiltration. Traditional GPU monitoring tools produce no alert for either case. Runtime-level observability is the only layer where these behaviors leave a trace.
How eBPF Reaches the CUDA Runtime Without Code Changes
eBPF runs sandboxed programs inside the Linux kernel without modifying application code or kernel source. Kang compares it to JavaScript added to HTML: the kernel becomes programmable. For GPU workloads, eBPF hooks into cudaLaunchKernel to track execution and cudaMemcpy to track data movement between host and device. The hooks work regardless of framework, PyTorch, TensorFlow, or a custom inference engine, because all of them pass through the same Linux kernel. The program deploys as a DaemonSet pod on every worker node.
Building a CUDA-Aware Grafana Dashboard in 10 Minutes
Kang assembled the demo dashboard the morning of the talk after seeing 700 registered attendees. The eBPF source code, written in C or Go, attaches BPF headers from lines 1 through 3 and exposes metrics to Prometheus. Grafana then visualizes CUDA-level error counts, kernel launch latency, and memory transfer rates alongside the standard DCGM hardware metrics. The combined view shows not just that utilization spiked but which application triggered the spike and at which runtime layer.
Ephemeral Containers for Safe Production Debugging
SSH access to worker nodes and kubectl exec both break container immutability and can violate compliance requirements. Ephemeral containers inject a temporary debugging container into the same pod, sharing its namespace. The ephemeral container can inspect the system in real time. When the debug session ends, the container disappears without touching the original application. Kang’s four-step workflow pairs DCGM and eBPF for detection, deploys an ephemeral container for detailed log inspection, analyzes the shared namespace, and resolves the issue with zero downtime.
Notable Quotes
GPUs don’t fail like CPU. So, they fail very silently. Brandon Kang · ▶ 00:25
this same blindness but also created some security risk. So, because if you because you cannot see what’s happening in the CUDA runtime level, you also can’t detect any malicious behavior inside your CUDA runtime. Brandon Kang · ▶ 08:20
I found out that the more than 700 attendees they registered to this session. So, uh I decided to make uh the dynamic demo for eBPF the the Grafana dashboard. But as I mentioned, it only takes uh it only took me uh 10 minutes. Brandon Kang · ▶ 21:43
instead of the modifying the running container, we inject a temporary debugging container into the same pod. Brandon Kang · ▶ 27:20
Key Takeaways
- DCGM reports hardware state but cannot identify CUDA runtime root causes.
- eBPF hooks into cudaLaunchKernel and cudaMemcpy without any application code changes.
- Unauthorized GPU kernel execution, including crypto mining, is only detectable at the runtime layer.
- A DaemonSet deployment pushes eBPF monitoring to every worker node without pod restarts.
- Ephemeral containers enable production debugging without breaking container immutability or compliance.
About the Speaker(s)
Brandon Kang is a Principal Technical Solutions Architect at Akamai Technologies, driving cloud computing presales engagements and providing strategic cloud-native consulting across Asia, including Japan, China, India, Australia, New Zealand, and Korea.