AI
The Problem: GPUs Sit Idle in One Cluster While Another Starves
AI workloads keep increasing while GPUs and memory grow more expensive and scarcer. Many organizations can no longer buy dedicated GPUs for every team or every cluster. Kubernetes manages hardware only inside each cluster boundary. There is no straightforward, safe way to lend devices across clusters. Idle capacity in cluster A cannot easily help cluster B. That boundary mismatch is the problem CoHDI targets.
Attach When Needed, Detach When Not
CoHDI keeps a shared GPU and memory pool. During the day, inference traffic is heavy, so more GPUs attach to inference clusters. At night, training and batch jobs dominate, so devices detach from the inference cluster and attach to the training cluster. The shift reduces idle time and lowers infrastructure cost. GPUs go where they create the most value at the right time.
CoHDI: Composable Hardware in Disaggregated Infrastructure
CoHDI stands for composable hardware in disaggregated infrastructure. Composable infrastructure creates software-defined bare metal systems. With CoHDI, GPUs can be dynamically attached and detached to Kubernetes nodes on demand. A resource pool holds GPUs, memory, and other devices. A management layer composes the right bare metal configuration at the right time. Hardware becomes flexible rather than fixed to a single cluster permanently.
Current Status and How to Try It
CoHDI works with the Kubernetes scheduler and device resource interface. The team tracks Kubernetes scheduler changes in version 1.36 as beta. CoHDI version 0.0.1.1 was just released. A composable infrastructure emulator ships with the project so anyone can test the attach and detach workflow without physical hardware. Code, docs, and contribution links are available via the QR code shown during the talk, and the team was present at the project pavilion.
Notable Quotes
GPU are allocated to where we they create the most value at the right time. Takao Indoh · ▶ 02:32
attach when needed and detach when not needed. Takao Indoh · ▶ 02:12
This make the hardware flexible and flexible not fixed. Takao Indoh · ▶ 03:31
Key Takeaways
- Kubernetes cluster boundaries block GPU sharing; CoHDI breaks that constraint with software-defined hardware.
- Dynamic attach and detach cycles match GPU allocation to inference or training demand by time of day.
- CoHDI 0.0.1.1 ships with an emulator; Kubernetes 1.36 scheduler integration is in beta tracking.
About the Speaker(s)
Takao Indoh is a Senior Director at Fujitsu. He has worked on Linux system support at Fujitsu since 2001, with a focus on crash dump frameworks including LKCD, diskdump, and kdump for mission-critical systems. He has also contributed to several open source communities around those projects.