AI
The Real Cost of Managing Policies Across Many Clusters
Platform engineers no longer manage two or three clusters. They manage many, across different zones and Kubernetes versions, with different rules per environment. Checking whether all production clusters meet security compliance today means jumping between screens, running kubectl commands, and reading policy reports by hand. A user at KubeCon asked Zhao how to verify that 20 policies across 100 clusters were actually functioning daily. There was no simple answer. That gap, between what GitOps shows as green and what is truly enforced, is what this talk addresses.
Three-Component Architecture: Kyverno, Open Cloud, and Cloud Hub
The solution combines three pieces. Kyverno runs as the admission controller, handling validation, image verification, mutation, and resource generation inside the cluster. It recently graduated in CNCF and now supports five policy types based on Common Expression Language, including validating and mutating policies that apply to any JSON payload or HTTP request. Open Cloud acts as the orchestration layer, translating natural language into Kubernetes actions and switching between local and remote clusters. Cloud Hub stores reusable skills, packaging senior-engineer knowledge into a shared pool that any user can pull from.
Live Demo: Installing Kyverno and Skills Across Two Clusters
Kuang ran the demo against two separate VMs, each hosting a kind cluster with one control plane and two worker nodes. Open Cloud deployed Kyverno on both clusters via Helm. When the remote cluster had no Helm CLI installed, the agent fell back automatically to kubectl apply without any manual intervention. After Kyverno was running, the agent cloned six Kyverno skills from a GitHub repository onto both nodes. Skills cover installing policies, auditing security status with the Kyverno CLI, and showing violations.
Policy Enforcement Verified in Real Time
With privileged pods already running in the local cluster’s default namespace, the agent scanned for pod security standard violations and returned a report broken down by violation type and affected resource. On the remote cluster it deployed four RBAC-related validating policies from predefined skill templates, then deployed a CRD deletion-protection policy and created test CR instances. When Kuang tried to delete the critical database CR, Kyverno blocked the request. A mutating policy test followed: the agent deployed a pod and confirmed that the runAsNonRoot security context was automatically patched on both clusters.
Security Guardrails for AI Agents on Kubernetes
Giving an AI agent cluster access introduces real risk. Kuang named identity confusion, supply chain attacks, and prompt injection as current concerns from security researchers. The recommended controls are isolation, network lockdown, sandboxed execution, audited skill libraries, and mandatory human approval for commands running on remote clusters. Every destructive or remote action in the demo required an explicit approval step before execution. Kyverno provides the policy rules. Open Cloud provides the orchestration. Neither replaces the human decision point for high-impact changes.
Kyverno Roadmap: SDK, Release 1.18, and ClusterPolicy Removal
Kyverno launched its SDK earlier this year, giving programmatic access to the core engine. All internal components, the controllers, CLI, and playground, use the same SDK, which guarantees consistent policy evaluation results. Release 1.18 targets a unified experience across the CLI and playground, covering all policy types including authorization policies. The reporting stack is migrating to the Open Reports project. ClusterPolicy was deprecated earlier this year. Removal is scheduled before KubeCon North America, targeting October. Teams that have not started migrating should start now.
Notable Quotes
you have to jump between different screens and type a lot of commands uh including cubecuddle uh just to figure out you know to access the logs the events and all the uh reporting system to see hey whether uh your cluster is you know security compliance or not Shuting Zhao · ▶ 02:00
bring AI agent to your Kubernetes cluster is just like give a robot the keys to your kingdom. If you don’t careful, it will go wrong fast. Dahu Kuang · ▶ 26:16
don’t worry the open cloud will auto fall back to use the coupl apply to continue the installation. Dahu Kuang · ▶ 18:04
keep the human in the loop is very important Dahu Kuang · ▶ 27:12
Key Takeaways
- Six Kyverno skills installed from GitHub cover policy install, audit, and violation reporting.
- Open Cloud requires explicit human approval before running commands on remote clusters.
- ClusterPolicy type will be removed before KubeCon North America; migration should start now.
About the Speaker(s)
Shuting Zhao is a Kyverno maintainer and Staff Engineer at Nirmata, the company behind Kyverno. She has mentored contributors through LXF mentorship programs since March 2021.
Dahu Kuang is Security Tech Lead on the Alibaba Cloud Container Service for Kubernetes (ACK) team, focused on container security and secure supply chain design and implementation.