AI
Scope and History of the Kubernetes Third-Party Audit Program
The Kubernetes SIG Security audit sub-project has run two full audits before this one. Atredis and Trail of Bits completed the first in 2019. NCC Group delivered the second in 2022. Those audits covered core components: kubelet, the API server, and related subsystems. All high-severity findings were fixed before each report went public. About half the findings from both audits remain open, and GitHub issue 18982, a certificate revocation gap for user identities, has been open for ten years.
Defining Scope for Non-Core Components
With 2.2 million lines of code, no single audit covers all of Kubernetes. This audit targeted non-core components. Rey Lejano and the team examined 665 features that graduated to stable between releases 1.24 and 1.31, then asked every SIG whether any area needed external review. That process produced 93 candidate features and areas. The team marked 17 as high priority. Final scope included Cluster API, pod security admission, and cgroup v2. OSTIF served as the independent third-party body that managed vendor selection and administration throughout.
Shielder’s Audit Methodology: Threat Modeling and Triage
Shielder used a hybrid divide-and-conquer approach. For each component, the team first asked three questions: where does untrusted input enter, is that input handled unsafely, and what assets are at risk if it is. Each component then went to a single auditor responsible for findings. Pietro Tirenna emphasized that findings had to be actionable. Flooding maintainer backlogs with a 200-page report of theoretical observations was a failure mode the team explicitly avoided. Bugs consistently appeared at integration points, not inside individual well-maintained components.
Three Vulnerability Classes Found Before Public Release
The first finding: the Kubernetes image builder ships a hardcoded Windows administrator password. Most cloud providers overrode that variable at build time. Nutanix did not. Every Windows node deployed on Nutanix via image builder was trivially accessible to any attacker with network reach. Shielder reported it immediately rather than waiting for the full report, and a CVE was issued. The second finding: AppArmor pre-bundles profiles that alias directly to the unconfined profile. An attacker on a cluster with pod security standards enabled at baseline could reference one of those profiles to run a fully unconfined container, bypassing the intended restriction.
Cluster API Supply Chain: Management Cluster Takeover
Cluster API lets administrators create workload clusters by applying a YAML template to a management cluster. Providers, including AWS, Azure, and VMware, supply those templates. Shielder found multiple paths where an attacker who can influence how a provider template is fetched or applied can inject an arbitrary privileged pod into the management cluster. The management cluster typically holds credentials for every workload cluster in the federation. Compromising it is equivalent to compromising all clusters it manages. Full details remain withheld until the public report releases.
Q&A
What was the most surprising finding? Pietro Tirenna pointed to OS-assumption violations, specifically a vulnerability tied to Windows that was unexpected in a Kubernetes context. Tabitha Sable noted that path-construction code written when Linux was the only Kubernetes platform has historically caused CVEs when applied to Windows paths. ▶ 24:10
Do AI tools help or hinder audit work? Pietro said AI accelerates understanding of unfamiliar codebases; taint analysis questions that previously took weeks now take minutes. Iain Smart added that a targeted question to a model against 2.5 million lines of Go returned the exact causative line in about three minutes, but said models cannot yet reliably find all bugs autonomously. ▶ 28:30
How does the Security Response Committee handle trust on a global open-source project? Tabitha Sable explained that SRC membership uses a private nomination process rather than a project-wide vote, allowing reputation built through prior community work to serve as the trust signal. ▶ 30:51
Notable Quotes
it’s important that we provide actual findings that can improve the security posture rather than just filling a 200 pages report with things that no one will ever read. Pietro Tirenna · ▶ 12:29
none of the issues that we found were due to um insecure coding or not following the best practices really like the code is very mature but the problem is that each developer of has the has visibility on their own project. Pietro Tirenna · ▶ 23:13
I’m not going to read two and a half million lines of Go. Not a chance. um the models were able to get through them in I think I was about three minutes before I got a reply with the exact line of code causing the behavior. Iain Smart · ▶ 29:52
genuinely means well and is capable of handling private disclosure information without tweeting about it or or whatever. Tabitha Sable · ▶ 32:05
Key Takeaways
- About half of findings from the 2019 and 2022 audits remain open and need attention.
- A hardcoded Nutanix image builder password received an emergency CVE before the full report published.
- Bugs in mature Kubernetes code appear at integration points, not inside individual components.
About the Speaker(s)
Iain Smart is a Principal Consultant at AmberWolf, co-lead of the Kubernetes audit sub-project, and previously delivered the 2022 NCC Group audit from the vendor side. He reviews cloud-native deployments and runs offensive security engagements.
Tabitha Sable is a Staff Engineer at Datadog and serves the Kubernetes community as co-chair of SIG Security and a member of the Security Response Committee. She writes exploits and hardens infrastructure.
Amir Montazery is Managing Director and Cofounder of OSTIF, a Chicago-based nonprofit focused on improving security for open-source projects. OSTIF served as the independent audit champion for this engagement.
Rey Lejano is a Solutions Architect at Red Hat and co-lead of the Kubernetes audit sub-project. He co-chairs SIG Docs and contributes to SIG Security, having served as Release Lead for Kubernetes 1.23.
Pietro Tirenna is a Security Researcher at Shielder, the firm that performed this audit. He conducts ethical hacking, penetration testing, and security research with a focus on complex software stacks.