AI
Why Single-Tenant Clusters Became a Security Bottleneck
Asana runs a work management platform trusted by over 170,000 organizations. For years the team operated single-tenant EKS clusters called “cube apps,” each owned by a separate infrastructure team. The infrastructure security team managed AWS IAM roles and security groups centrally for all of them. That centralization made security reviewers the sole deployment gate. A simple permission change could take days, not minutes, because engineers had to wait for review and then coordinate deployment timing with a separate security infrastructure rollout.
The Scaling Break: From Cube Apps to Cells
Single-tenant clusters also created single points of failure. A misconfigured security group could take down Asana entirely. Each team had to become a Kubernetes expert, solving the same operational problems repeatedly. The replacement architecture, called cells, uses multiple multi-tenant EKS clusters. Each cell serves only a subset of customer traffic, so a failure stays contained. Dedicated platform teams own the Kubernetes layer, freeing application engineers to focus on business logic rather than cluster operations.
Pod-Level Permissions: IAM and Networking Colocated With the App
In the old model, IAM used instance profiles scoped to the whole cluster. If application A needed S3, application B got S3 access too. Network security was similarly the union of every workload’s permissions. The fix was shrinking the permission unit to the pod. Crossplane composite resource definitions now express IAM policies alongside the deployment YAML, packaged with Helm and deployed via Argo CD. Cilium network policies default to deny, so every allowed flow must be declared explicitly. A new DynamoDB table permission ships in the same PR as the code that calls it, cutting the old one-to-two-day cycle to a single review.
Automation for Generating Least-Privilege Policies
Rewriting all permissions for multi-tenancy was an opportunity to automate. Hubble, Cilium’s logging tool, provides L7 observability showing which pod attempted which DNS name. AWS client-side monitoring captures every IAM permission an application tried to use. The team collects these logs over a 30-day window and converts them into Cilium network policy and Crossplane IAM policy definitions. Developers start with a baseline scope, then tooling generates the least-privileged YAML for them. The same log pipeline feeds a SIEM for real-time behavioral verification.
Kyverno as the Invisible Guardrail
The platform layer adds its own controls. Bottlerocket provides a minimal OS on nodes. Istio handles all inbound and outbound cluster traffic, enforcing mTLS by default inside the cluster. Kyverno acts as the runtime gatekeeper, rejecting specs with excessive privileges before they enter the cluster. The team moved Kyverno’s cluster policies into local linting so engineers catch violations on their laptops, not after a deployment. When AI agents generate Kubernetes resources, those same policies run in the agent’s context, holding generated YAML to the same security standard as human-authored specs.
Lessons: Guardrails Must Fail Loudly and Ship Early
Invisible guardrails stay invisible until someone breaks them. When the first workload needing SQS access was added to cells, the application was correctly configured but the permission boundary never included an SQS allow statement. The result was silent failures and debugging sessions. The fix: keep boundaries permissive on the specifics, strict only on properties that truly matter, like preventing deletion of data-bearing resources. Abstractions can hide too much. Label-based network policies confused developers who could not find the policy; a single explicit network policy file next to the deployment YAML solved that.
Q&A
What stops a compromised application B from creating a Crossplane IAM role that grants itself access to application A’s S3 bucket? Crossplane always attaches the permission boundary, so the fabricated role cannot exceed the maximum scope, though the team acknowledged the protection is limited for cross-app read access. ▶ 24:17
Why ship per-app network policies rather than reusing Cilium global network policies? Global policies are too opaque. Developers miss them, assume connectivity works, then hit failures they cannot diagnose. Asana keeps global policies to a strict minimum (observability agents only) and gives each application one explicit policy file next to its deployment YAML. ▶ 25:16
How does decommissioning a resource work if permission boundaries block deletions? Deletions tied to cluster lifecycle are allowed. For data-bearing resources like RDS, deletion requires an operator action that is logged and tracked. User-facing applications never hold delete rights on those resources. ▶ 26:36
Do Kyverno policies go straight to enforce mode? The team starts in audit mode briefly, then the platform team fixes all violations across the monorepo and switches to enforce, typically within a day. ▶ 27:59
Notable Quotes
ask ourselves the question, what if the most secure path was the fastest one? James Elías Sigurðarson · ▶ 05:03
we aren’t just giving them guardrails and telling them to do their best, but we’re giving them a GPS to pave the path for them, which makes the secure path also the fastest path. James Elías Sigurðarson · ▶ 14:27
you are not your user. Vignir Hafsteinsson · ▶ 21:05
if it were to be compromised, hopefully it will never be, but if it were, you could basically destroy the company. James Elías Sigurðarson · ▶ 27:29
Key Takeaways
- Colocating IAM and network policies with deployment YAML cut permission change lead time from days to one PR.
- A 30-day log window from Hubble and AWS client-side monitoring generates least-privileged policy YAML automatically.
- Kyverno policies running as a local linter catch over-privileged specs before they reach the cluster, including AI-generated ones.
- Permission boundaries cap the maximum scope any workload can hold, blocking data-deletion rights from user-facing apps.
- Label-based network policy abstractions confused developers. One explicit policy file per app, next to the deployment YAML, fixed that.
About the Speaker(s)
James Elías Sigurðarson is an Infrastructure Engineer at Asana, specializing in building scalable, resilient, secure, and developer-friendly platforms. He believes the best infrastructure is the kind you don’t have to think about, and has spent several years building toward that goal from Asana’s Reykjavik office.
Vignir Hafsteinsson is an Engineering Manager for one of Asana’s platform teams, also based in Reykjavik. He brings experience leading infrastructure projects across banking, gaming, and biotech, and applies both technology and organizational culture as tools to empower engineering teams.