Diverse User Needs on Shared Kubernetes Infrastructure

▶ Watch (0:03)

CERN’s LHC is a 27 km ring that produces terabytes of data per second. Physicists working in the control room have different requirements: one wants an H100 GPU session, another needs a shell with physics software pre-installed, a third wants Jupyter notebooks, and a fourth needs distributed training across nodes. To serve all of them on shared compute, CERN built a Kubernetes platform using CNCF tools and published a reference architecture to the CNCF a few days before this talk.

Heterogeneous Hardware and Storage Layout

▶ Watch (2:58)

The cluster mixes Nvidia H100s, H200s, and L40s, AMD MI300X and Radeon Pro GPUs, plus CPU nodes with high memory or high core count. InfiniBand nodes use ConnectX-7 NICs; others use RoCE v2. NVMe scratch volumes handle high-IOPS compilation, Longhorn provides per-user shared PVCs, and Ceph handles data extraction outside the cluster. Container images run 30-40 GB for standard ML work and up to 150 GB for FPGA simulation, so popular images are pre-pulled across nodes to cut startup time.

SSH Access and CPU Topology Pinning

▶ Watch (5:56)

Users get SSH access via ContainerSSH, a CNCF project that uses pod names as identifiers. Authentication comes from an OIDC token or a Kerberos ticket. By default, Kubernetes spreads container processes across CPU cores and NUMA nodes, causing cross-NUMA memory latency. With the topology manager enabled in the kubelet, processes get pinned to exclusive cores with local NUMA memory access. Chiorescu ran a live stress test: 46 cores allocated, 384 threads launched, all constrained to the allocation without affecting neighboring pods.

Full MLOps Stack from Notebooks to Model Serving

▶ Watch (11:15)

The platform covers interactive development, distributed training, hyperparameter tuning, and model serving. KubeFlow Notebooks provide PyTorch, TensorFlow, and SciPy environments accessible via browser, SSH, or VS Code attached to a running cluster session. Distributed training uses KubeFlow and Ray Train SDKs, declared as PyTorchJob specs. A live demo launched one master and five workers, each using six Nvidia H100 NVLink GPUs, to reconstruct collision particles. Pre-built pipelines for model quantization and FPGA synthesis let users submit jobs by pointing at a model pickle file.

GPU Direct RDMA over InfiniBand

▶ Watch (15:31)

Standard Ethernet interconnects stage GPU-to-GPU transfers through host memory, adding CPU overhead. CERN uses InfiniBand and RoCE v2 NICs, configured with the Nvidia Network Operator and GPU operators. These enable GPU Direct RDMA: GPUs communicate directly across InfiniBand without touching host memory or the CPU. A bandwidth benchmark across two H100 SXM GPUs on separate physical nodes reached 50 GB/s. The software stack runs on UCX and MPI for integration with user workloads.

Kyverno Policies and Platform Adoption Lessons

▶ Watch (19:25)

Kyverno mutation policies auto-mount home directories, shared volumes, and RDMA configuration; validation policies block users from overriding their own resource allocations. This collapses a complicated pod spec into a short label-annotated spec. Adoption required exposing familiar interfaces: SSH and the same storage systems physicists have used for 20 years. During the ramp-up period, the team released updates daily and iterated on rapid user feedback. CERN upstreamed a new ContainerSSH operating mode that shipped days before this talk.

Q&A

Have you considered KubeVirt instead of pods running sleep-infinite with 40 GB images? The team acknowledges the tradeoff but says current storage capacity handles it and they will revisit if sizes grow further. ▶ 24:22

How many active users does the platform have? Hundreds, drawn from CMS, Atlas, and the theory department collaborating on the same platform. ▶ 24:54

Do you have auditing or threat detection in place given the level of user freedom? Falco is deployed and triggers alerts on suspicious behavior; Kyverno validation policies prevent privileged container access. ▶ 26:02

Is cloud-native intended to replace classic SLURM or PBS-based HPC? The goal is to work alongside HPC, using Interlink and potentially Slinky to bridge the two worlds. ▶ 27:42

Notable Quotes

Yeah, it’s probably the Wi-Fi. Okay, Raulian-Ionut Chiorescu · ▶ 9:36

like we already paid for them so yeah Raulian-Ionut Chiorescu · ▶ 27:04

Key Takeaways

  • CERN’s Kubernetes ML platform serves hundreds of physicists across CMS, Atlas, and the theory department.
  • The topology manager pins processes to exclusive CPU cores with local NUMA memory access, eliminating cross-NUMA latency.
  • InfiniBand RDMA delivers 50 GB/s GPU-to-GPU bandwidth without CPU or host memory involvement.
  • Kyverno mutation and validation policies collapse complex pod specs into simple label-annotated manifests.
  • CERN upstreams platform changes to avoid long-term maintenance of local forks.

About the Speakers

Raulian-Ionut Chiorescu is a Computing Engineer at CERN, working within the Kubernetes team on Machine Learning Operations. Before CERN, he worked as a DevOps Engineer at an AI company in Cambridge.

Hannes Hansen is a computing engineer at CERN working on machine learning operations on Kubernetes. Before CERN, he helped develop the grid data management for the experiments.