AI
The Configuration Problem Hiding Inside Every AI Cluster
Running a GPU training workload on EKS with an H100 requires over 250 configuration values spread across 15-plus components. Switching that same cluster from training to inference means swapping five components and setting 40-plus additional values. NVIDIA’s own team manages accelerated Kubernetes infrastructure for workloads spanning world model simulation to robotics, and they hit this complexity every day. The tuned configuration that makes clusters perform well has stayed locked inside NVIDIA, and this project is the first step to change that.
ACRE: Encoding Cluster Knowledge as Recipes
Each ACRE recipe is a version-locked set of components paired with explicit dependencies, manifests, and optimized Helm values files tuned for a specific use case. Recipes include a constraints block that lists the minimum cluster requirements before deployment can proceed. After deployment, built-in validators check component readiness and can confirm conformance against the CNCF AI conformance program, including generating the artifacts needed to submit a PR for certification. More inputs at recipe creation time produce a more tightly optimized output for the target environment.
Querying and Generating Recipes Offline
ACRE ships both a CLI and a self-hosted REST API for air-gapped data centers. The tool encodes its full knowledge graph inside the binary, so pipelines can query it in real time without any external connection. To create a recipe, operators supply the cloud service, GPU type, intent (training or inference), and OS. The output writes to either a local filesystem or an OCI image, keeping the artifact portable and storable in any standard registry.
Bundle Attestation and Supply Chain Provenance
Once a recipe exists, ACRE generates a bundle of artifacts that Argo CD or Helm can deploy directly. ACRE itself is self-attested at build time, which means every bundle it produces carries a verifiable provenance tied to the exact ACRE version that generated it. A bundle stored in GitHub or GitLab can be re-validated months later to confirm it was not tampered with. Additional deployer integrations are in progress beyond the current Argo CD and Helm support.
Initial Recipes and Community Contributions
NVIDIA seeded ACRE with recipes for H100 and GB200 accelerators running on EKS and GKE. Within one week of the project opening publicly, Microsoft Azure and Oracle both contributed recipes. The repository at github.com/nvidia/acre accepts contributions from hyperscalers, AI cloud providers, and individual users. NVIDIA’s goal is to grow the set of recipes across hardware types and environments beyond what the internal team can cover alone.
Notable Quotes
we’ve been doing a number of investments across Nvidia. We’ve been working at projects like the DRA driver, the GPU operator, Kai scheduler. Nathan Taber · ▶ 00:17
There’s over 250 different configuration values that you have to make across 15 plus components. Nathan Taber · ▶ 00:55
because ACRE encodes the entire graph of that knowledge inside of the binary, you can actually query that graph in real time in your CI/CD pipeline without connection to the outside world. Mark Chmarny · ▶ 00:227
we’ve seen contributions from both Microsoft Azure and Oracle. Nathan Taber · ▶ 00:368
Key Takeaways
- A single H100 EKS training workload requires 250-plus configuration values across 15-plus components.
- ACRE recipes are version-locked, constraint-validated, and ship with CNCF AI conformance checks built in.
- Bundle attestation ties every deployment artifact back to the exact ACRE binary version that produced it.
About the Speaker(s)
Nathan Taber is Head of Product for AWS Kubernetes and has been on the launch teams for several AWS container services. He currently sets the vision and direction for Amazon Elastic Kubernetes Service and works closely with AWS customers on Kubernetes at scale.