AI
Why Per-Tenant KEDA Beats a Shared Instance
KEDA supports namespace-scoped TriggerAuthentication resources, which makes a shared instance look safe at the object level. The problem sits in the authentication flow. When KEDA uses federated credential exchange, the bootstrap token that starts the chain belongs to the operator process. If that operator is shared across tenants, the trust boundary is shared too. Preferred Networks runs clusters across multiple organizations, so a shared trust boundary violates their core requirement: failures and credentials must never become another tenant’s problem.
The Clusterwide External Metrics API Problem
Kubernetes HPA uses a single clusterwide API group, external.metrics.k8s.io. Only one backend can be registered per API group. A second registration simply overrides the first. Installing a separate KEDA metrics server per tenant does nothing if there is no way to route requests to the right one. The HPA resource carries a metric name but no backend selector. The fix: keep one APIService as the front door and place an NGINX router behind it. The external metrics URL path already contains the namespace, so the router extracts it with a regular expression and forwards to the correct per-tenant backend without any custom code.
Rebuilding the TLS Trust Chain Through the Router
Placing a proxy in the aggregated API path breaks authentication. The KEDA metrics server expects a client certificate from the Kubernetes API server, whose CA is distributed via the extension-apiserver-authentication ConfigMap. When the NGINX router terminates TLS and opens a new connection upstream, the backend sees the router as the caller. Authentication falls through to the anonymous user group and the SubjectAccessReview is denied. The repair requires three changes: the router presents its own client certificate, the metrics server trusts the router CA instead of the default, and the router preserves X-Remote-User and X-Remote-Group headers without rewriting them.
Dynamic Namespace Scope With a Namespace Reloader
Each per-tenant KEDA operator should watch only the namespaces belonging to its tenant. KEDA provides a WATCH_NAMESPACE environment variable, but it accepts only a static comma-separated list. The controller-runtime library underneath does not allow changing the watch scope after the manager starts. Preferred Networks solved this with a wrapper process called the namespace reloader. It watches for tenant namespace updates, maintains the environment variable, and restarts the operator when the set changes. The KEDA deployment manifest stays close to upstream, requiring only an image swap and a few injected environment variables to keep upgrade costs low.
Network Policies That Follow Tenant Boundaries
Default Kubernetes has no network isolation between namespaces. A misconfigured or malicious pod in tenant A can reach tenant B’s metrics server, operator, or user workloads directly. Preferred Networks classified all traffic into scopes (tenant-system, tenant-wide) and identified allowed paths explicitly. Cross-tenant traffic is denied. Kubernetes built-in NetworkPolicy is namespace-scoped, requiring a policy per namespace and making it hard to scale. Cilium’s cluster-wide network policy lets rules span an entire organization using namespace-level selectors. Because they use HNC and namespaces change frequently, cluster-wide policy is the only practical option.
Full Architecture and Design Lessons
The finished design keeps one APIService front door and one shared admission webhook cluster-wide. Per tenant: one operator, one metrics server, one namespace reloader. Open-source tools are not built for any specific platform, so adaptation requires glue code or operational workarounds. When evaluating a shared service for a multi-tenant platform, Preferred Networks checks three things: operations (ease of management at scale), reliability (blast radius and noisy-neighbor risk), and security (permissions, network paths, and authentication methods). Strong isolation and low operational cost pull in opposite directions, and the right configuration follows from actual business requirements.
Notable Quotes
the boost drop tokens that starts the chain belong to the operator process. Aya Igarashi · ▶ 09:46
If you try to register a second one, it’s simply overridden. Aya Igarashi · ▶ 11:37
No custom needed. It’s all in the URL. Aya Igarashi · ▶ 12:40
Security is balanced between strong isolation and operational costs. Aya Igarashi · ▶ 24:32
Key Takeaways
- A shared KEDA operator means a shared trust boundary for federated credential exchange.
- One NGINX router with namespace-aware URL routing solves the single clusterwide metrics API constraint.
- Cilium cluster-wide network policy scales tenant traffic isolation without per-namespace duplication.
About the Speaker(s)
Aya Igarashi is a Software Engineer at Preferred Networks, Inc. She has worked on Kubernetes-based platform development since 2016, focusing on cloud-native open-source projects. She also co-organizes the Kubernetes Meetup Tokyo.