AI
Why Policy Enforcement Fails in Practice
Pods running as root in production, discovered during an incident rather than a policy violation, is the opening problem Pal puts to the audience. Policy enforcement sounds simple on paper. At scale it does not stay that way. Running multi-tenant apps can mean hitting 10,000 policies in a single cluster. HIPAA, PCI, and GDPR auditors do not accept “we have a policy” as an answer. They want the exact audit log showing what was audited, when, and what decision was made. Without that trail, the audit fails.
Selection Parameters That Actually Matter
Two days of CEL versus two to four weeks on another policy language is a day-and-night difference in team adoption. Beyond learning curve, Pal flags the API server’s 30-second global timeout across all webhook requests. Every mutating webhook, every validating webhook, every custom webhook adds to that budget. Exceed it and deployments fail in production. He also recommends starting with failure mode set to ignore, proving the engine is stable, and only then switching to blocking. Day-two questions like HA, upgrade paths, and DR need answers at day zero.
Kubernetes-Native Validating Admission Policy
CEL is baked into the API server. No extra webhooks, no external dependencies, no network calls. The failure question shifts from “what if my webhook pod crashes” to “what if my API server crashes,” which is a rarer but cluster-wide event. Policies use two objects: one defines the rule, one binds it to a namespace. That lets teams roll out gradually, enforcing on staging first and expanding to production later. Mutating Admission Policy arrives GA in Kubernetes 1.36, closing the mutation gap for straightforward defaults like adding resource requests.
Kyverno: YAML Policies With Background Scanning
Kyverno’s main selling point is that teams already writing Kubernetes YAML face almost no new learning curve. Its four policy types, validate, mutate, generate, and image verification, cover use cases that native CEL cannot. Generate is particularly powerful: creating a new namespace automatically copies a network policy from a source namespace and keeps it synchronized if the source changes. Image verification uses cosign to confirm signatures and check digest integrity, so operators know exactly what bytes run in production. Background scanning catches policy violations on existing workloads without requiring a resource update.
OPA Gatekeeper: Cross-Stack Policy With Rego
OPA and Gatekeeper are separate projects. OPA is a general-purpose engine that speaks Rego and works outside Kubernetes entirely. Gatekeeper is the Kubernetes integration, adding CRD-based constraint templates and audit capabilities on top. A central security team writes constraint templates; platform teams instantiate them with their own parameters. That split gives centralized governance with distributed control. Cross-resource validation, checking whether an ingress hostname conflicts with existing ingresses, is a concrete Rego capability that CEL cannot replicate. The trade-off is memory: every synced resource type increases Gatekeeper’s memory footprint.
Decision Tree and Real-World Recommendations
Pal’s decision tree starts with two questions: do you need mutation or resource generation, and do you need image signature verification? Answering yes to either points toward Kyverno. If simplicity is the priority and neither is needed, native CEL wins. OPA Gatekeeper pays off only when policy enforcement cuts across multiple infrastructure layers, Terraform, microservice authorization, and multi-cloud. For a startup platform team, Kyverno is the starting point. For a security-focused fintech, a dual layer of native CEL for simple checks plus Kyverno for advanced cases is the prescription. Kubewarden suits teams that already have systems programming experience and need Wasm isolation.
Notable Quotes
Start in audit mode. Don’t go full-blown [snorts] uh block right away. Start with audit, make sure everything is fine, then only tighten things down. Nabarun Pal · ▶ 35:01
There is a 30 secondond limit total on all web hook requests that can be made as part of a request response flow. And this is not just for one single webbook. This will be total of all web hooks. Nabarun Pal · ▶ 07:13
open policy agent is probably more than you need. Uh the investment is real when you choose open policy agent and if you’re not getting the cross tech cross tag benefit it probably will not pay off over time. Nabarun Pal · ▶ 30:41
day two things actually should be thought on at day zero and decision should be made accordingly. Nabarun Pal · ▶ 06:30
Key Takeaways
- Start with Kubernetes-native CEL; add Kyverno or Gatekeeper only when CEL falls short.
- The API server enforces a 30-second total budget across all webhook requests; plan accordingly.
- Always deploy in audit mode first, confirm compliance reports look correct, then block.
About the Speaker(s)
Nabarun Pal is a Principal Software Engineer at Broadcom and a maintainer of the Kubernetes project. He is an emeritus member of the Kubernetes Steering Committee, chairs Kubernetes SIG Contributor Experience, and serves as a Release Manager for Kubernetes.