AI
Why Static GPU Allocation Breaks at Scale
SK Telecom’s H cluster launched around July 2024 with over 1,000 Blackwell GPUs, built with Penguin Solutions and Super Micro. Each HGX node ships with eight GPUs statically allocated per team. A team needing only two GPUs for inference still consumes a full node. That fragmentation raises TCO and leaves GPU utilization low. Without virtualization, provisioning delays compound the problem: end users unfamiliar with driver installation can stall deployments by days, and multi-tenant workloads expose security risks when the full infrastructure remains visible to all.
Choosing Containerized VMs Over OpenStack or Plain Containers
OpenStack spawns VMs with a guest kernel sandbox that protects neighbors if one VM is compromised, but its performance is poor and it lacks Kubernetes extensibility. Plain containers are fast and extensible via CRDs but lack the isolation of a guest kernel. SKT chose containerized VMs using KubeVirt, running it alongside Kubernetes so VMs are orchestrated like containers. KubeVirt did not target AI workloads natively, so SKT spent significant effort optimizing the stack before running it in production with Blackwell hardware.
NVLink Virtualization and GPU Direct RDMA
Each Blackwell SXM node interconnects eight GPUs through an NVSwitch at up to 1.8 TB/s bidirectional. Keeping that bandwidth inside a VM requires two steps. First, NVSwitch partitions are managed via Nvidia Fabric Manager to allow or block GPU-to-GPU links per tenant. Second, GPU Direct RDMA requires enabling ATS (Address Translation Services) on the HCA so that incoming NIC traffic bypasses system memory and lands directly in GPU memory. Without ATS, IO-MMU forces all traffic through the CPU and system memory, collapsing performance. SKT also wrote a PCI topology sidecar container that rewrites the libvirt domain XML at runtime to mirror the physical PCIe topology inside each VM, ensuring NIC-GPU pairs share the same PCIe switch.
Benchmark Results: Near Bare-Metal Performance
With topology alignment and ATS in place, SKT measured NVLink bandwidth on H100 at roughly 700 GB/s bidirectional inside a VM, matching bare metal. On Blackwell, the virtualized NVLink result overlaps bare metal with less than 1% degradation. Relying on PCIe alone instead of NVLink produces roughly 1,700% worse bandwidth. NCCL single-node tests show the VM and bare-metal curves completely overlapping. The 1,000-GPU scalability test, run on a six-spine, 24-leaf topology with 64-port NDR switches and eight-rail optimization, confirms that performance degradation stays acceptable as GPU count grows.
InfiniBand Tenant Isolation Without Performance Loss
Hard multi-tenancy requires isolating the GPU fabric per tenant. For RoCE networks, standard VLAN or VxLAN handles this. For InfiniBand, SKT uses partition keys, analogous to VLAN IDs. HCAs sharing the same partition key can communicate; all other traffic is blocked. SKT built a Kubernetes operator that watches VM events. When a VM joins a GPU cluster, the operator queries cluster policy, discovers the VM’s HCA via Nvidia UFM, and enforces the correct partition key automatically. NCCL all-reduce benchmarks across 16 GPUs show no measurable throughput difference with or without partitioning enabled.
Observability Stack for 1,000-GPU Operations
Each VM ships with two pre-installed agents: a node exporter and a DCGM exporter. Metrics flow to Prometheus, which fires alerts on anomalies to Slack, email, or other channels. Nvidia’s XID error codes let operators write precise alerting rules for GPU failure types. SKT built a GUI where each rectangle represents one GPU node containing eight GPUs, with utilization and health visible at a glance. The full observability layer runs on CNCF projects and requires no manual intervention from end users managing their workloads.
Notable Quotes
the performance wide the degradation is quite negligible. It’s almost like one less than 1% degradation. Jian Li · ▶ 20:04
without properly uh virtualizing the MV link which means you only relying on PCIe for example communication then uh compared with that MV link you can outperform almost like a 1,700% performance improvement you can you can observe Jian Li · ▶ 20:13
our end user do not really claim about the performance at all because it’s almost similar to the bare metal Jian Li · ▶ 26:44
Key Takeaways
- Virtualizing NVLink via Fabric Manager partitions keeps intra-node GPU bandwidth within 1% of bare metal.
- ATS on the HCA and a PCI topology sidecar are both required to run GPU Direct RDMA inside a KubeVirt VM.
- A Kubernetes operator enforcing InfiniBand partition keys delivers hard multi-tenancy with no NCCL throughput penalty.
About the Speaker(s)
Jian Li is Principal Research Engineer at SK Telecom with extensive experience in AI infrastructure virtualization and product management. He leads the architecture and development of Petasus AI Cloud, SK Telecom’s in-house platform for heterogeneous AI workloads supporting containers, VMs, and managed Kubernetes across Nvidia, Intel, and AMD accelerators.