AI
Why AI Traffic Patterns Break Flat Scheduling
LLaMA-style dense models concentrate GPU communication within a single network rail, so leaf switches handle most traffic locally. DeepSeek-style mixture-of-experts models route data to different experts, generating all-to-all traffic that crosses spine switches between rails. PD disaggregation for inference requires prefill and decode pods to share KV cache over RDMA, so they must land on topologically close nodes. In a multi-tenant cluster treated as a single pool, workloads that scale without topology awareness cause network congestion and unfair bandwidth allocation between tenants.
Why Node Affinity Fails at This Scale
Node affinity with pod affinity rules requires operators to manually map physical topology to static labels. When the cluster grows, static partitions create conflicts between applications and operational costs rise. Labels carry no awareness of live infrastructure state: if a cable is removed or a switch goes under maintenance without a label update, the scheduler operates on a stale map. Node affinity is also set-based matching with no hierarchical view. Gang scheduling without a global topology picture can produce partial allocation and resource deadlocks across the cluster.
Three Schedulers That Enforce Topology Constraints
Kueue acts as a gatekeeper: it calculates remaining capacity inside each topology domain and injects node selectors into the pod template before handing off to the native scheduler. Volcano replaces the default scheduler entirely, using HyperNode CRs to build a logical topology chain, and integrates directly with the InfiniBand UFM to auto-discover network wiring. Kai Scheduler, from Nvidia, adds subgroup support inside a pod group so that prefill and decode pods each carry independent topology constraints while still respecting a group-wide placement rule. All three express topology as a hierarchy: node, super pod, rack, and spine.
Automatic Topology Discovery vs Manual Labels
Every advanced scheduler depends on accurate per-node topology labels. Wrong labels produce wrong placement decisions. Nvidia Topograph connects to Spectrum, Quantum, and NVSwitch fabrics, continuously builds an accurate topology map, and writes labels for every hierarchy level automatically. For non-Nvidia environments, a custom topology builder can query switches running Sonic OS via gNMI to collect neighbor tables, and use each GPU vendor’s CLI or API to read domain IDs. Both paths remove the human from the labeling loop and keep labels synchronized with actual physical wiring.
Cost-Effective Validation Without GPU Hardware
Validating scheduler behavior or a custom topology builder on real GPU clusters with multi-layer switching is prohibitively expensive. KWOK (Kubernetes Without Kubelet) generates thousands of virtual nodes inside a kind cluster within seconds. Each virtual node can be labeled with topology attributes like spine or rack and configured with arbitrary GPU counts. Pods scheduled to KWOK nodes immediately report as running with no containers started, so the entire test operates at the control plane level. This lets teams deploy Kai Scheduler, submit simulated inference workloads, and inspect placement decisions at no hardware cost.
Nvidia Air for Full Switch Simulation
KWOK covers scheduler validation but cannot test a topology builder that must actually talk to switches. Nvidia Air is a free platform that deploys Linux VMs and virtual Ethernet switches running Sonic OS or Cumulus OS. A full multi-layer topology can be built exactly as desired, with Kubernetes deployed on top. The custom topology builder runs inside that cluster, queries the virtual switches, and generates hierarchical node labels. Nvidia Air also exposes APIs to programmatically create infrastructure from user-defined topology configs, which makes building CI pipelines for automated topology builder testing straightforward.
Notable Quotes
it’s no longer enough to uh just find the free slots we must find the right slots. Weizhou Lan · ▶ 04:14
If the labels are wrong the scheduling decision will be wrong. Weizhou Lan · ▶ 15:19
My answer is cork. It is it mean it means kubernetes without kubate. Weizhou Lan · ▶ 20:55
My my practical solution is Nvidia air. Nvidia air is a free s platform to simulating network infrastructure of data center. Weizhou Lan · ▶ 22:55
Key Takeaways
- DeepSeek MoE all-to-all traffic and PD disaggregation both require topology-aware pod placement to avoid spine congestion.
- Static node affinity labels drift from physical reality and cannot represent hierarchical topology or coordinate gang scheduling.
- KWOK generates thousands of labeled virtual GPU nodes in seconds, making scheduler validation free of physical hardware.
About the Speaker(s)
Weizhou Lan is a Senior Tech Lead at DaoCloud with over 15 years of engineering experience, including 8 years working with Kubernetes. His focus covers networking, eBPF, service mesh, and AI infrastructure. He has delivered 5 KubeCon talks and 3 KCD talks.