AI
Why the Default Kubernetes Stack Falls Short for ML Workloads
When CERN onboarded users onto their Kubeflow platform, three questions came back immediately: why is my training slow, how much energy am I consuming, and can I have more GPU. The default Kubernetes stack, node metrics and cluster dashboards, could not answer any of them. It is built for microservices, not ML workloads. That gap is what pushed the NGT team to build their own observability layer on top of Kubeflow, covering GPU telemetry, energy attribution, and idle resource tracking.
The Observability Architecture: Exporters, Prometheus, and Grafana
Physical nodes run multiple exporters in parallel. Nvidia and AMD GPU exporters feed hardware telemetry. IPMI and Kepler handle energy. The standard node exporter and cAdvisor cover CPU, memory, and container metrics. Kubeflow itself exports job lifecycle and notebook controller data. Prometheus, deployed via the kube-prometheus stack, scrapes all of them. Grafana sits on top for visualization. Alert Manager watches idle sessions and fires emails or Mattermost notifications when users leave GPU sessions sitting unused.
Energy Monitoring and CO2 Accounting per Workload
IPMI gives node-level power readings from physical hardware components. Kepler goes further, attributing energy to individual pods and workloads. It uses eBPF kernel modules and sensors to collect CPU package, core, and memory subsystem data, then maps that back through the container and pod hierarchy. Because CERN’s data center runs on the French electrical grid, fetching the public carbon intensity API, measured in grams of CO2 equivalent per kilowatt-hour, and multiplying by power consumption produces a per-workload CO2 figure. That feeds directly into CERN’s sustainability reporting.
Scaling Prometheus to 2.3 Million Series
Running all those exporters is expensive. The team crashed their Prometheus deployment several times and lost data before tuning it. Their current configuration holds around 2,300,000 concurrent series. They added retention policies, switched to PVCs so restarts no longer wipe data, and set up remote write to long-term storage outside the cluster. Remote write requires filtering, because sending every series is not feasible. The team also learned that new AMD CPUs can break Kepler’s eBPF module if the kernel version is too old, and that MIG GPU partitions behave differently from full GPUs.
What the Dashboards Revealed: A CPU Governor Story
Three Grafana dashboards cover user workloads, cluster-wide capacity, and sustainability. The user dashboard scopes all metrics by namespace. One user running quantum lattice field simulations saw benchmark scores fluctuate on identical hardware. GPU utilization was 100%, memory was fine, CPU usage looked normal. The CPU frequency panel showed the answer: the Linux CPU governor was varying clock speed between runs. Fixing it required a kernel-level change. Without the frequency panel, the cause would have remained invisible. The idle session panel on the cluster dashboard recovers GPU capacity by prompting users to release unused sessions.
A Proposed Upstream Kubeflow Observability Manifest
Building this stack from scratch takes weeks. To fix that, the team opened an upstream Kubeflow issue proposing a single observability manifest. The manifest would ship a sane Prometheus config, all needed ServiceMonitors, pre-built Grafana dashboards, and configuration for Kepler and DCGM GPU metrics. The goal is to go from zero to full observability in under five minutes. They asked attendees to scan a QR code and add a thumbs-up to the issue. Future work includes Kepler power monitoring for VMs, which currently only works on bare-metal nodes, and Kubeflow service-level metric dashboards.
Q&A
What is the minimal Grafana dashboard to start with on Kubeflow? The proposed upstream manifest is the recommended starting point, because it bundles the metrics and panels that ML workloads actually need, unlike the default microservice-oriented stack. ▶ 22:17
How do you balance maximizing GPU utilization with minimizing power draw? Higher power does not always mean better results. If power consumption rises faster than benchmark gains, it is not worth it, and physicists comparing AMD, Nvidia, and FPGA accelerators need the efficiency ratio, not raw throughput. ▶ 23:28
Notable Quotes
power aware AI is no longer optional. We need to think about the sustainability and we cannot just build nuclear power plants with each data center Amine Lahouel · ▶ 20:17
we cached our cluster uh our Prometheus uh multiple times and we even lost the data and we learned some hard lessons along the way Amine Lahouel · ▶ 12:02
go from zero to all this observability in possibly less than five minutes Laura Llinares · ▶ 19:00
if the power consumption is increasing more than the the gains in benchmarks, it’s not worth it Amine Lahouel · ▶ 24:14
Key Takeaways
- The default Kubernetes stack cannot answer GPU, energy, or efficiency questions for ML workloads.
- Kepler attributes energy consumption down to the pod level using eBPF kernel modules.
- Multiplying IPMI power readings by the French grid’s public carbon intensity API gives per-workload CO2 figures.
- 2.3 million concurrent Prometheus series required retention policies, PVCs, and filtered remote write to avoid data loss.
- A CPU governor bug causing benchmark fluctuations was only visible through a CPU frequency panel in Grafana.
- An upstream Kubeflow observability manifest could compress weeks of setup into under five minutes.
About the Speaker(s)
Amine Lahouel is a Software Engineer at CERN with nearly a decade of experience, focused on making MLOps accessible across the NGT scientific computing platform.
Laura Llinares is a DevOps Engineer at CERN. She holds a double Master’s degree in Computer Engineering from Politecnico di Torino and Cybersecurity and Cloud Computing from Paris. Her work at CERN includes dynamic GPU allocation and the sustainability dashboards shown in this talk.