AI
Why Standard Kubernetes Conformance Falls Short for AI Workloads
Kubernetes was designed for stateless, CPU-bound web services. AI workloads bring GPU-intensive compute, strict scheduling topology requirements, and high-bandwidth networking demands that standard conformance never tested. Vendors filled the gap with their own solutions, producing fragmentation across distributions. The “write once, run anywhere” promise breaks for AI. Janet Kuo announced the Kubernetes AI Conformance program at KubeCon NA last year to set a common capability baseline that platforms must meet to run AI workloads reliably.
The Six Pillars of AI Conformance
The program defines six areas a platform must cover. DRA and accelerator support is the foundation. Observability means accelerator metrics are exposed. Scheduling covers gang scheduling and autoscaling. Networking focuses on the inference gateway. Security is its own pillar. Finally, platforms must support deployment of complex AI operators. Most certified distributions today are commercial offerings. kOps aims to be one of the first fully open-source, community-driven distributions to meet all six.
How kOps Has Run Conformance Since Day One
kOps has existed since Kubernetes 1.0 and has run the standard Kubernetes conformance program from its start. Scalability tests run up to 5,000 nodes on Google Cloud and AWS through kOps, with results published in the open. The project runs on AWS primarily but also supports GCE, Azure, Hetzner, and OpenStack. All test runs go through Prow, funded by cloud credits donated to the CNCF by AWS, GCP, Azure, Oracle, and Nvidia. The project spends several million dollars a year running those tests.
Fast-Path Tests and the Automated Submission Pipeline
kOps is a volunteer project with limited sponsorship, so the team built one automated fast-path test for every AI conformance requirement. The reference configuration uses the Nvidia GPU operator and Prometheus stack on AWS. Tests run every few hours through Prow, emit YAML results in the format the AI conformance program expects, and automation submits those results upstream to the Kubernetes AI conformance sub-project. The conformance project then runs its own approval automation against that YAML file, making the full pipeline end-to-end automatic.
Upstreaming Tests to Prevent Fragmentation
Each distribution writing its own private tests recreates the fragmentation the conformance program exists to prevent. The goal is to upstream the kOps test suite into a centralized framework that any distribution, including Cluster API, can run without modification. A new cloud provider or sovereign-cloud operator can read the reference configuration, stand up a cluster, run the shared automated tests, and get a conformance result. This covers training, inference, fine-tuning, and distillation workloads under a single shared tooling baseline.
Q&A
Are other cluster lifecycle tools moving toward AI conformance? Justin Santa Barbara, as SIG Cluster Lifecycle lead, said the team is actively upstreaming the kOps test suite so tools like Cluster API can reuse it, and encouraged users to push their providers to support it. ▶ 19:49
Does AI conformance actually verify real behavior rather than just API surface? Justin confirmed AI conformance is much harder to pass without real AI capability, and noted that even non-AI users benefit because the tests verify cluster autoscaler, HPA, and other infrastructure that every cluster needs. ▶ 21:09
Notable Quotes
we spend several million dollars a year just running tests for Kubernetes projects Justin Santa Barbara · ▶ 12:35
it is much harder to pass it without actually being able to do real AI Justin Santa Barbara · ▶ 21:18
healthy opensource makes managed Kubernetes also better. Arnaud Meukam · ▶ 08:09
we basically want to like automate at maximum to secure confirance verification Arnaud Meukam · ▶ 15:09
Key Takeaways
- Standard Kubernetes conformance does not require persistent volumes or GPU support; AI conformance does.
- kOps built one automated fast-path test per AI conformance requirement, tied to the Nvidia GPU operator and Prometheus stack.
- The team is upstreaming all tests into a shared framework so Cluster API and other distributions can reuse them without duplication.
About the Speaker(s)
Arnaud Meukam is an open-source engineer and core Kubernetes contributor with over five years in the project. He chairs the Kubernetes Infrastructure SIG and serves as a release manager.
Janet Kuo is a Staff Software Engineer at Google Cloud who has contributed to Kubernetes since before the 1.0 launch in 2015. She is a Kubernetes maintainer and leads SIG Apps as chair and tech lead.
Justin Santa Barbara has contributed to Kubernetes since 2014. He was the primary maintainer of Kubernetes AWS support and started the kOps project. He currently leads SIG Cluster Lifecycle at Google.
Ciprian Hacman is a software engineer at Microsoft specializing in Kubernetes and platform engineering. He has contributed to and maintained Kubernetes projects for eight years and currently leads the etcd-operator working group.