AI
The Problem: Kubernetes Resource Quotas Stop at the Namespace Boundary
Kubernetes ships a native ResourceQuota object, but it is namespace-scoped. A team spread across multiple namespaces has no single place to see total consumption, and platform admins have no built-in way to cap that team’s combined spend. The alternatives — scripts, manual tracking, or a hand-rolled controller — all require ongoing maintenance. The talk frames three requirements: visibility across namespaces, enforcement that lets developers self-serve inside a team budget, and full automation when new namespaces appear.
KRO and Kyverno: What Each Tool Does
KRO (KubeRow) is a continuously running operator that takes a Resource Graph Definition (RGD) and generates a CRD from it. Instances of that CRD trigger KRO to create and manage the child resources defined in the RGD. Kyverno is a policy engine deployed as an admission controller. It intercepts API server requests and evaluates CEL expressions to accept or reject them. Together, KRO handles state aggregation and object lifecycle while Kyverno handles enforcement at admission time.
Defining the ClusterResourceQuota with an RGD
The RGD has two sections. The schema section defines the ClusterResourceQuota CRD fields: team name, a budget map for CPU and memory hard limits, and a status block. The status block uses CEL expressions to sum all ResourceQuotas that carry the team’s label, producing a live total-usage dashboard. The resources section tells KRO to manage two child objects per ClusterResourceQuota: a read-only external-ref watcher that finds labeled ResourceQuotas, and a Kyverno ValidatingPolicy scoped to that same label selector.
Demo: Blocking an Over-Budget CPU Request
After applying the RGD, KRO exposed the ClusterResourceQuota CRD. Crowder applied a ClusterResourceQuota for a production team with a 10-CPU budget, then added two namespace-scoped ResourceQuotas. KRO tracked both and summed them in the status field. When Crowder tried to edit one namespace quota to request 20 CPU, Kyverno blocked it immediately. The error message read that CPU requests would bring the total to 21 against a budget of 10. No operator code was written; only a few lines of YAML and CEL were needed.
Q&A
How do you stop teams from stealing quota by forging another team’s label? This demo trusts teams to apply labels correctly, but additional Kyverno policies can enforce label values, and external RBAC systems can lock label mutations. ▶ 25:25
Is KRO production-ready, and how close is 1.0? The features in the demo shipped in the 0.9 release on Tuesday; the project has load-tested managing millions of resources and has 18 active enhancement proposals, but remains pre-GA with possible breaking changes before 1.0. ▶ 26:09
Could native Kubernetes admission controllers replace Kyverno here? Yes for the basic budget check, but Kyverno adds CEL library functions that can fetch existing ResourceQuotas inside the policy itself, removing the dependency on KRO’s aggregated status. ▶ 27:53
Notable Quotes
Kyverno is a policy engine for Kubernetes. Mariam Fahmy · ▶ 05:52
CPU requests will bring the total to 21 with this edit, but our budget’s only 10. Adam Crowder · ▶ 23:40
some of the features I use in this demo were only with 0.9 release that we released like on Tuesday. Adam Crowder · ▶ 26:15
We’ve load tested it with like millions of managing millions of resources at once. Adam Crowder · ▶ 27:30
Key Takeaways
- One RGD definition is written once; every new team needs only one ClusterResourceQuota manifest.
- KRO’s external-ref watcher aggregates ResourceQuota totals across namespaces automatically via label selectors.
- Kyverno rejects over-budget ResourceQuota changes at admission time before any object is created.
About the Speaker(s)
Mariam Fahmy is a System Engineer at Cloudflare and a maintainer of Kyverno, the Kubernetes-native policy engine. She has contributed to Kyverno’s evolution, including new CEL-based policy types, and mentors contributors through the CNCF LFX program.
Adam Crowder is a senior software engineer at AWS working on EKS. He has been involved with the KRO project since its inception and champions its adoption inside Amazon and across the Kubernetes community.