AI
Why Namespace Multi-Tenancy Breaks Under Best-Practice Hardening
ERNW researchers asked one question: does industry best-practice hardening actually isolate namespace-based multi-tenancy? The answer, found across real customer engagements, was no. Namespace multi-tenancy is attractive because it costs less than running separate clusters, but hardening it correctly is harder than most platform teams expect. The team found multiple isolation breaks that best practices do not cover, and disclosed three of them at this talk.
Exploit 1: Stealing Kubeflow Session Cookies via Cross-Namespace VirtualService
Kubeflow, a CNCF machine learning platform, assigns every user a namespace and a default-editor service account. That account holds permission to create Istio VirtualServices. A tenant can write a VirtualService in their own namespace that references the Kubeflow gateway in a different namespace, override the dashboard’s favicon route, and forward requests to an attacker-controlled pod. Every user who loads the Kubeflow dashboard sends cookies through that pod. The attacker gets full session control over every Kubeflow user in the cluster. Kubeflow fixed the issue by removing the VirtualService create permission from the service account.
Exploit 2: Borrowing mTLS Secrets via Traefik Annotation References
Traefik’s SericesTransport CRD lets operators configure how the proxy connects to a pod, including mutual TLS credentials. A tenant with permission to create a Kubernetes Service can add an annotation pointing to a SericesTransport resource in a different namespace, then forward traffic to their own pod. Traefik’s proxy performs the mTLS handshake on the attacker’s behalf, so the attacker uses the victim’s client secret without ever reading it directly. The impact can extend beyond the cluster: the same annotation pattern lets a tenant reference GCP compute certificates that belong to other projects in the same cloud account. Traefik updated its security documentation and recommends migrating to the Gateway API.
Exploit 3: Istio Service Mesh Traffic Interception Across Namespaces
Sven Nobis demonstrated the data-plane break. An attacker deploys a VirtualService bound to the service mesh (not a gateway) and claims a host, for example example.com. The Istio sidecar in a victim’s pod sees that VirtualService and routes traffic to the attacker’s pod instead of the internet. This is a man-in-the-middle attack at the mesh layer. Istio maintainers consider the behavior expected, a tradeoff made before multi-tenancy existed in Kubernetes, and changing it would be a breaking change. Their recommendation is to use the Gateway API. ERNW and Istio jointly published a security blog post on the day of this talk.
A Three-Step Methodology to Find and Fix These Problems
The methodology has three steps: detect whether you actually use namespace multi-tenancy, assess which resources a tenant controls, and address what you find. Detection is harder than it sounds. CI/CD pipelines running pull-request code in the same cluster as production secrets, or applications with customer scripting capabilities, both create implicit multi-tenancy. Assessment covers both control-plane interactions (what can a tenant create via the Kubernetes API?) and data-plane interactions (can a tenant reach a component’s service or sidecar?). Use the CIA triad to score each interaction against confidentiality, integrity, and availability of other tenants or the platform.
Addressing Findings: Vendor Fixes, Gateway API, and Admission Policies
When a vendor fix exists, apply it. When migration to the Gateway API is possible, it handles cross-namespace references securely by design. When neither option is available immediately, admission policies fill the gap. The Kyverno policy library contains policies that block several namespace multi-tenancy escape patterns. For cluster-specific problems, write custom admission policies. Two concrete examples: block tenants from deploying VirtualServices to the service mesh entirely, or whitelist only the hostnames that belong to a given tenant and reject any reference to gateways or hosts outside their namespace.
Notable Quotes
we found some problems or multiple problems which are not well studied yet. Lorin Lehawany · ▶ 01:14
the attacker can lock those requests and they have access to the cookies of the cubeflow user. Lorin Lehawany · ▶ 08:54
So this is a classic man and middle attack. Sven Nobis · ▶ 17:06
the IST maintainers considers this issue expected behavior. It’s a preferable experience tradeoff because this APIs were designed even before airbug in Kubernetes existed. Sven Nobis · ▶ 18:14
Key Takeaways
- Kubeflow’s
default-editorservice account let any tenant steal every user’s session cookies via a cross-namespace VirtualService. - Traefik’s annotation-based
SericesTransportreferences exposed mTLS client secrets to tenants without giving them direct secret read access. - Istio considers cross-namespace service mesh interception expected behavior; migrate to the Gateway API to close the gap.
About the Speaker(s)
Lorin Lehawany is a security analyst at ERNW Enno Rey Netzwerke GmbH specializing in penetration testing and Kubernetes security. She improves infrastructure security for companies across Germany and is an active member of the BlackHoodie community.
Sven Nobis is a senior security researcher and analyst at ERNW Enno Rey Netzwerke GmbH with more than 10 years in IT. His work centers on cloud security, security assessments, red teaming, and consulting for well-known companies in Germany.