AI
EU Sovereign Cloud: What the Requirements Actually Mean
EU sovereign cloud means more than data residency. The stack must satisfy GDPR compliance, local data processing, and models trained on sovereign data. LLM chatbots and AI agents must keep data within EU boundaries. The IPCEI program (Important Projects of Common European Interest) backs this with roughly 8 billion euros to build a multi-cloud, multi-provider cloud-to-edge continuum. SAP created the Neoforce Foundation to coordinate participating organizations under this mandate.
The Infrastructure Stack: From Bare Metal to Kubernetes
The stack starts with a metal automation layer using Kubernetes declarative APIs to manage compute, network, and storage. Above that sits IronCore, an IaaS variant that extends Kubernetes extension API servers to handle virtualization (via libvirt, QEMU, KVM), block and object storage (Ceph), and accelerated packet processing through DPDK. Garden Linux, a lean hardened Debian-based distribution, runs on servers, switches, smart NICs, and GPU hosts. It ships the latest LTS kernel with repeatable auditable builds, AppArmor, SELinux, and fully signed kernel modules for secure and trusted boot.
Garden Linux GPU Flavor: RDMA and GPU Direct Out of the Box
For the AI reference stack, SAP built a Garden Linux flavor that ships RDMA, GPU Direct RDMA, and RoCE support by default. Kernel modules included are DMA-buf, RDMA-specific modules, Mellanox modules, and proprietary NVIDIA and NVIDIA-UVM modules, with a planned migration to open-source NVIDIA kernel modules for Turing and newer architectures. IOMMU and memlock settings are tuned for RDMA. Network interfaces use jumbo frames with MTU above 9,000. The NVIDIA Network Operator sets up SR-IOV for direct hardware passthrough, enabling true zero-copy, kernel-bypass, GPU-to-GPU communication at full line rate.
Gardener and Grove: Managed Kubernetes with Hierarchical Scheduling
Gardener runs as a Kubernetes-as-a-service product, deployable on-premises across AWS, Azure, GCP, and OpenStack providers. Shoot cluster control planes are bin-packed into seed clusters via a component called Gardenlet, the same conceptual role as kubelet but for clusters. This eliminates dedicated VMs per control plane. For the AI reference stack, a Gardener extension called “AI Acceleration” automatically installs the NVIDIA GPU Operator, Network Operator, Dynamo, Grove, and Kai scheduler. Grove, built jointly by SAP and NVIDIA, provides a PodClickSet API for hierarchical gang scheduling across heterogeneous multi-node workloads.
Grove and Dynamo: Disaggregated Inference Orchestration
Dynamo splits inference into prefill, decode, and front-end components. Prefill builds the KV cache; decode reads it and generates tokens. NCCL libraries in the image optimize KV cache transfer over NVLink or RDMA fabric. A KV-aware router in the front end maximizes cache reuse. Grove maps this to a PodClickSet, grouping PodClicks into scaling groups that scale proportionately. The Dynamo Planner integrates into Grove as an SLA-based horizontal autoscaler reacting to real-time metrics. For GB200 multi-node NVLink architectures, Grove automatically generates compute domains and manages NVSwitch IMX channels without user intervention.
Demo Setup: Four B200 Nodes Running Mistral Large
The lab uses four NVIDIA B200 nodes. Each node has BlueField-3 DPU NICs for north-south traffic at 2x100 GB and eight ConnectX-7 NICs at 400 GB each for RoCE. A single NVIDIA Spectrum-X switch provides 800 GB Ethernet. The reference application is Mistral Large NF4 in disaggregated mode: one front-end PodClick, one prefill worker requesting 8 GPUs (one full node), and one decode worker requesting 8 GPUs. Startup takes 5 to 7 minutes as init containers load model weights. The recorded demo shows front-end requests per second increasing with time-to-first-token stabilizing around 30 milliseconds.
Notable Quotes
it essentially stabilizes somewhere around 30 millconds. Sanjay Chatterjee · ▶ 31:15
we are trying to do a live demo on a lab setup. So, let’s hope everything works. Madhav Bhargava · ▶ 19:11
true zero copy kernel bypass GPU to GPU communication at line rate full line rate. Sanjay Chatterjee · ▶ 09:34
Key Takeaways
- Garden Linux ships RDMA, RoCE, and GPU Direct support as a first-class Gardener extension.
- Grove’s PodClickSet API handles hierarchical gang scheduling and topology placement across B200 multi-node clusters.
- The IPCEI program commits roughly 8 billion euros to a multi-cloud EU sovereign cloud-to-edge continuum.
About the Speaker(s)
Sanjay Chatterjee is an engineering manager at NVIDIA focused on GPU scheduling to scale AI and HPC workloads on Kubernetes. His research covers novel programming models, parallel languages, and runtime systems.
Madhav Bhargava is a development architect at SAP and a maintainer of the open-source Gardener project. His research covers distributed systems, runtime systems, scheduling, scaling recommender systems, and language runtimes.