Kubernetes as a Bidirectional Distribution Engine

▶ Watch (1:13)

Macleod describes Kubernetes as the narrow waist of the infrastructure hourglass. Write to the Kubernetes APIs from a framework like Ray or Slurm and you get access to every infrastructure provider and hardware vendor. A hardware provider that integrates with Kubernetes becomes available to every framework and service built on top, at no additional cost. It functions as a bidirectional megaphone. The 200-plus CNCF projects are the result of that two-way amplification. That density is what makes Kubernetes a distribution engine in both directions.

Three Phases: Container Orchestration to Distributed OS for AI

▶ Watch (1:53)

Kubernetes evolved in three phases. Phase one brought container orchestration, beginning with stateless applications and expanding to stateful and batch workloads. Phase two produced the 200-plus CNCF projects visible today. Phase three treats Kubernetes as a distributed operating system for AI. Operating systems shift focus to resource management and frameworks that let different workload types share the same hardware. Heterogeneous hardware is now accelerating that shift.

Topology-Aware Scheduling and Hardware Support in 1.36

▶ Watch (3:08)

Kubernetes 1.36 introduces a workload API with topology-aware scheduling. Gang scheduling places an entire job as a unit rather than scattering pods across the cluster. Workload-aware preemption can evict lower-priority work when resources are scarce. DRA (Dynamic Resource Allocation), which reworked how Kubernetes talks to hardware, is largely complete. An agent sandbox announced in November adds safety boundaries for agentic workloads running at scale. The scheduler will gain deeper understanding of hardware topology in the next phase.

Open RL and Fine-Tuning on Kubernetes

▶ Watch (4:33)

Google’s Gemma models have been fine-tuned across domains including medical applications and translation. Teams wanting to fine-tune on sensitive data need local infrastructure rather than external API services. Thinking Machine Labs’ Tinker provides a simple fine-tuning API that Sky RL also adopted. GKE Labs’ Open RL builds on this pattern: develop a training loop on a laptop and deploy it on Kubernetes in minutes. Hundreds of similar tools will emerge each week. The platform needs to absorb all of them without requiring long-term bets from teams.

Notable Quotes

megaphone. And it’s really interesting. Jago Macleod · ▶ 1:25

But that’s not the point. The point is Jago Macleod · ▶ 5:32

that lets you experiment and innovate. Jago Macleod · ▶ 5:39

Key Takeaways

  • Kubernetes 1.36’s workload API adds gang scheduling and topology-aware preemption for heterogeneous hardware.
  • DRA reworked how Kubernetes talks to hardware; the agent sandbox covers agentic workloads at scale.
  • Open RL lets teams develop training loops locally and deploy on Kubernetes in minutes.