SR-IOV: Hardware-Level Network Virtualization for Kubernetes

▶ Watch (0:38)

SR-IOV is a PCIe standard that lets a single physical NIC expose multiple logical interfaces called virtual functions (VFs). Each VF appears to the OS as an independent PCI device with its own MAC address and PCI configuration space. The NIC handles this in hardware, bypassing the kernel and enabling deterministic low-latency packet processing without CPU involvement. All VFs share the physical NIC’s bandwidth, which makes bandwidth planning a central concern. Use cases include telco CNFs, HPC scientific computing, and distributed AI/ML training with large inter-node data volumes.

A Production RDMA Deployment at Expo Osaka 2025

▶ Watch (3:52)

At the 2025 Expo in Osaka, NTT ran video inference workloads from onsite cameras, covering fire detection and crowd congestion detection. The pipeline connected the pavilion to a regional edge cloud over RDMA using Nvidia ConnectX with SR-IOV virtual functions. The site-to-site link ran over an APN photonics network through an optical-electrical convergence switch. Kanda noted the same RDMA inference pipeline applies to standard deployments without APN, making this architecture broadly relevant beyond the Expo setup.

DRA and the Kubernetes Network Driver Model

▶ Watch (5:00)

KEP 3698 originally captured multi-network goals including VF support and bandwidth guarantees. That KEP closed, but the work continued through Dynamic Resource Allocation (DRA). Two DRA APIs matter most here: consumable capacity (KEP 5075) and partitionable devices. A new Kubernetes network driver model bridges CNI and device plugin concepts. It uses DRA for scheduling and NRI for configuration. Drivers become composable and independent, each knowing when to attach, how to configure, and when to release resources in step with pod lifecycle events.

Bandwidth Planning With Partitionable Devices: Live Demo

▶ Watch (14:14)

The demo ran on two worker nodes: Eagle04 with an Intel E810 and Eagle05 with a ConnectX-6. A config map assigned 80% bandwidth to VF0 and 20% to VF1 on Eagle04, and 40%/60% on Eagle05. The CND driver reflected those percentages into the resource slice: 80k Mbps (80 Gbps) and 60k Mbps (60 Gbps). Pod A requested a VF with allocated bandwidth above 50 Gbps via a CEL selector and received Eagle05 VF1. Pod B, making the same request, received Eagle04 VF0, the remaining qualifying VF. IP configuration matched inside each container.

KEP 5941: Dynamic Bandwidth Requests and Shared Capacity

▶ Watch (20:59)

The partitionable devices approach requires cluster admins to preconfigure every VF before deployment. End users pick from that fixed pool rather than specifying the bandwidth they need at deploy time. KEP 5941, authored by Sebastian (present at the talk), extends consumable capacity in two ways. First, multiple devices can share one underlying capacity pool, covering VF bandwidth, shared IP pools, L3 cache, and memory bandwidth. Second, it adds optional capacity with guarantees, so best-effort workloads and guaranteed-bandwidth workloads can coexist on a node without overprovisioning or breaking each other’s reservations.

Q&A

Does the DRA CNI driver replace the existing CNI or sit on top of it? The driver calls CNI internally, passing the CNI config inside the resource claim, so any CNI can be used alongside it. ▶ 29:05

Do pods need Multus to attach multiple interfaces in this model? In the demo, no Multus was used. The CNI config lives directly in the resource claim or device class. JNET does not use CNI at all. Whether Multus is needed depends on the specific DRA driver implementation. ▶ 32:28

Can hardware queue counts be configured through resource claims? Queues are shared across VFs. With KEP 5941, configuring queue counts when specifying a VF should be possible, and sub-functions from SR-IOV capabilities are also under consideration. ▶ 33:44

Is the end goal for VFs to be requested like CPU and memory directly in the pod spec? Yes. With KEP 5941, a pod should be able to request a VF and a specific bandwidth in one step, without selecting from a preconfigured pool. ▶ 35:03

Notable Quotes

the bandwidth is just example. So you can uh apply the various uh capacity uh things about about this with this new cap I think. Masaharu Kanda · ▶ 33:35

we don’t use moltus um the CNI config is directly in the resource flame or in the vice class and we don’t need multus at all. Lionel Jouin · ▶ 32:28

it allows workloads to coexist efficiently without overprovisioning hardware uh and breaking guarantees uh the bandwidth guarantees for example. Lionel Jouin · ▶ 23:00

the same kind of RDMA based inference pipeline is applicable in much more common deployments even without APN. Masaharu Kanda · ▶ 04:44

Key Takeaways

  • SR-IOV VFs share the parent NIC’s physical bandwidth, making bandwidth planning a scheduling concern, not just a hardware one.
  • The partitionable devices API treats each VF as an individual device, enabling CEL-based bandwidth selectors in resource claim templates.
  • KEP 5941 extends consumable capacity so pods can request VF bandwidth dynamically at deploy time rather than from a fixed preconfigured pool.

About the Speaker(s)

Lionel Jouin is a Software Engineer at Red Hat based in Stockholm, Sweden, and a former Ericsson Engineer. He is the multi-network project lead, a CNI maintainer, and has contributed to projects including Network Service Mesh.

Masaharu Kanda is a Senior Research Engineer at NTT. He began his career in electrical and embedded software engineering, working on Blu-ray-compatible optical drives before contributing to cloud gaming server development. He now develops the disaggregated computing infrastructure controller at NTT.