AI
What Interlink Does: Kubernetes Pods to Slurm Jobs
Interlink sits between a Kubernetes cluster and a Slurm-managed supercomputer. You write a plain pod manifest. Interlink translates it into a Slurm batch job on the remote HPC system. No custom resource definitions. No root or administrative network privileges. The Kubernetes cluster can run anywhere: a laptop, a cloud VM, or even a plane running K3S. The HPC center can be in the same building with network segregation or geographically distant.
Three Components, One Virtual Node
Interlink has three core parts. First, a virtual kubelet provider installs on the Kubernetes cluster. Second, an interlink server runs on the HPC side, either on a login node or a dedicated edge machine. Third, a network plugin lets offloaded pods communicate back to the internal pod network of the Kubernetes cluster. Once those are in place, you target the virtual node in your pod spec the same way you would any real node.
Requirements Before You Deploy
A reasonably recent Kubernetes version is the baseline. For full capabilities, an ingress with wildcard support helps. On the HPC nodes, a container runtime must be present and the file system must be shared across nodes. The edge node where the interlink server installs needs the unshare command available for network handling. Those are the hard requirements. The container runtime and shared file system are the constraints most HPC centers will need to verify first.
Demo: Kubeflow Notebook on the Leonardo Supercomputer
The live demo ran against Leonardo, a supercomputer at an Italian HPC center. Ciangottini launched a Kubeflow notebook, set the target node to the virtual node backed by Leonardo, and submitted. On the Slurm side, a job appeared on the login node. Logs were accessible from the Kubernetes web interface. When the notebook became ready, it had direct access to Leonardo’s GPUs. The same approach works for pipelines or any other pod-based workload, not only notebooks.
Notable Quotes
no CRDs plain pods definition and no root administrative network privileges to do all the the magic around Diego Ciangottini · ▶ 00:46
we are really talking about something that is remotely access slurm based resources from your own kubernetes that can run on your laptop Diego Ciangottini · ▶ 01:23
Hope you like this picture because it cost me to to make it works on the plane with the Wi-Fi Diego Ciangottini · ▶ 01:47
Key Takeaways
- Interlink translates plain pod specs into Slurm jobs with no CRDs or root privileges.
- Three components: virtual kubelet on Kubernetes, interlink server at HPC edge, network plugin.
- Any pod-based workload, including Kubeflow notebooks, can target remote GPU supercomputers.
About the Speaker(s)
Diego Ciangottini is a physicist who earned his PhD from the University of Perugia, Italy in 2012. He works as a technologist at INFN (Italian National Institute for Nuclear Physics), researching cloud-native solutions for the institute’s scientific use cases.