GPU Utilization Below 20%: The Cost of Exclusive Allocation

▶ Watch (0:15)

Production GPU clusters running the Nvidia device plugin in exclusive mode routinely sit below 20% utilization. Li showed two screenshots taken from a real production environment to make this concrete. Each GPU is locked to one workload whether or not that workload needs the full device. The wasted capacity is not a corner case. Li called it a very natural scenario for clusters that never adopted GPU sharing.

How Hami Shares GPUs Across Workloads

▶ Watch (0:39)

Hami partitions physical GPUs so multiple tasks run on the same device simultaneously. User A and User B can each pin tasks to a shared GPU pool and leave remaining capacity for other consumers. Two modes are available: standard Hami and the more advanced Hami DI. Both let operators specify GPU slices through ordinary Kubernetes resource limits rather than a ResourceClaim, which reduces the YAML a developer must write.

Hard Isolation via a Custom CUDA Library

▶ Watch (1:48)

The core of Hami is a self-implemented CUDA hacking library injected into each container at runtime. When a workload requests 10 GB of device memory, the library caps what nvidia-smi reports inside the container to exactly 10 GB. That cap is enforced by the library itself, not by a soft quota. The result is hard resource isolation: one container cannot consume memory allocated to another, even though both share the same physical GPU.

Installation and Minimum Requirements

▶ Watch (2:12)

Installing Hami requires fetching the Hami Helm repo and running helm install. After that, operators label the GPU nodes they want Hami to manage with gpu=on. The minimum requirements are Nvidia driver version 440 and Kubernetes 1.18. Using Hami DI raises the Kubernetes floor to 1.34. Li described the requirements as easy to satisfy for most clusters already running GPU workloads.

Volcano Integration and Workload Spec

▶ Watch (2:54)

Hami’s scheduling logic has been merged into the Volcano main branch. Enabling the device share plugin in Volcano hands scheduling to Volcano while Hami core handles in-container resource enforcement. A workload spec can request two GPUs, each with 10 GB of memory and 30% of total compute, using standard resource limits fields. No custom resource claim is needed. The split between Volcano scheduling and Hami enforcement keeps both components focused on what they do best.

Notable Quotes

these two pictures are taken in a real production environment. You can see that the overall environment of certain GPU is about below 20% Mengxuan Li · ▶ 00:26

The core component of Hami is a self-implemented uh CUDA hacking library which is injected into the container. Mengxuan Li · ▶ 01:52

in China, we have to make every GPU counts. Mengxuan Li · ▶ 04:03

Key Takeaways

  • Exclusive GPU allocation leaves clusters below 20% utilization in real production environments.
  • Hami enforces hard memory limits inside containers by injecting a custom CUDA library at runtime.
  • A single Helm install and a node label are all that is needed to start sharing GPUs with Hami.