The Fragmented Authorization Landscape

▶ Watch (1:00)

Companies run 1,000 to 4,000 microservices. Each service builds its own access control. Authorization logic lives in code, config files, or ad hoc solutions. Teams reinvent the wheel. Security cannot see who has access to what. Broken access control became OWASP’s top vulnerability in 2021. Attackers target authorization logic bugs because scanners cannot find them across distributed points. This fragmented approach also slows developer velocity as teams struggle to understand access requirements.

Decoupling Authorization from Application Code

▶ Watch (8:21)

The solution decouples authorization from application code entirely. A single control plane called Guard acts as the source of truth for all access policies. Guard is multi-tenant. Each use case is a tenant. It supports Topaz sidecar for distributed enforcement. Requirements: high availability (auth is in the critical path), efficient scaling with traffic, and flexibility from coarse-grained to field-level granularity. P99 latency targets milliseconds.

Architecture: Sidecar, Central, and Platform Evaluators

▶ Watch (16:55)

Authorization has three components: enforcement point (application), decision point (Topaz sidecar), and administration point (Guard). The sidecar model keeps the data plane independent. After initial policy download, decisions run on last known data. A data broker service processes over a million authorization decisions per second with 10-20 millisecond latency. For platform-level needs, policy is transpiled to Istio and Kubernetes native formats, achieving near-zero availability degradation.

Lessons from Rolling Out Guard

▶ Watch (27:19)

Localhost authorization calls added 1-2 milliseconds, measurable for high-traffic services. Adding a decision cache in the SDK cut latency from milliseconds to tens of microseconds. The cache expiry is set shorter than the sidecar’s periodic refresh, preserving security properties. Another lesson: Topaz denies everything until its initial policy sync completes. A container startup ordering fix: wait for Topaz to receive policies, prevents false denials. This fix was contributed upstream.

Driving Adoption and Long-Term Benefits

▶ Watch (30:40)

Adoption strategy: enable level-one granularity (API and endpoint) by default on the service mesh for all microservices. For new projects, recommend Guard during security review. Target existing risky services that lack granular access control. Easier to convince. On the platform layer, interfacing with one or two teams gives fleet-wide visibility. Long-term benefits include a single pane of glass for access, standard governance for access requests, just-in-time access, unused permission cleanup, and instant threat detection when a compromised identity probes resources.

Q&A

How are policies maintained across thousands of microservices? Level one granularity via Istio YAML/UI for API and method, level two via Guard multi-tenant self-service where teams own their policies. ▶ 36:51

Which authentication does Guard require? It is agnostic; humans authenticate through the IDP (e.g., Okta) and machines use API keys, mTLS, or JWTs. ▶ 38:08

What is the one thing to tell a team getting started? Do not build your own authorization solution; use the existing system. ▶ 40:33

How to adapt new policies to mimic old auth patterns? Run old and new decisions side-by-side on a dashboard, fix mismatches until no differences remain, then roll out. ▶ 40:52

Notable Quotes

In uh 2013, broken access control was not even uh in the top five of the risks. But since 2021, it has been the uh number one vulnerability Ashwin Sidhalinganahalli · ▶ 6:50

Uh authorization is nonoptional. If you can’t authorize, then the only secure fallback is to deny everything. Fletcher Ramee · ▶ 17:54

Um, do not build your own authorization solution. Please use the existing system. Ashwin Sidhalinganahalli · ▶ 40:42

Key Takeaways

  • Decouple authorization from application code to centralize policy management and improve security visibility.
  • Use Topaz sidecars for low-latency, resilient authorization; cache decisions to drop latency to microseconds.
  • Standardize authorization primitives across the organization to reduce developer cognitive load and accelerate incident response.

About the Speaker(s)

Ashwin Sidhalinganahalli is a Principal Security Software Engineer on the Production IAM team at Roblox. He blends theory and software engineering to build practical security solutions.