AI
Two Years, Nearly 400 Reviews, NPS 52
Over two years, Asana’s product security team ran nearly 400 security design reviews on large feature launches before they hit production. Roughly 150 received a full deep review. The program’s NPS landed at 52. The NPS creators classify anything above 50 as excellent. Apple’s overall NPS sits around 61, for products developers actually enjoy using. Asana’s security team earned a comparable score while running a process most developers consider a chore, which says something about how the intake experience was designed.
Why Self-Reporting Creates Coverage Gaps
Before the program was formalized, teams had to know security did reviews, judge their own risk correctly, and contact someone on the security team. Most didn’t. Coverage was a function of awareness and confidence, not actual risk. The fix: automation watched every new feature added to the roadmap and sent the security team a task to assess risk. Full coverage appeared, but the cost was immediate. Security engineers spent time triaging hundreds of features instead of doing threat models, with limited context for most of them.
Shifting Risk Triage to Engineering Teams
The second revision added a structured intake form. Every new roadmap feature triggered a task sent to the project team, not security. The form collected design docs and asked yes-or-no questions about common risks: outbound network requests, third-party integrations, custom crypto schemes. Any yes answer auto-triaged the feature for security review. Manual triage for security disappeared. The miss: “yes to any risk” was not granular enough. Features with low actual risk flooded the queue. Security team fatigue shifted from triage to a large influx of reviews.
Adding a Tech Lead to the Triage Gate
The current version inserts one step between the developer survey and security triage. A tech lead in the relevant feature area receives the survey results and decides whether the project warrants a review. That person knows the context: whether a similar threat model ran recently, whether outbound requests in this area actually change the risk profile. Security sees only reviews a domain expert already flagged as new. Variability in those judgments is the tradeoff, and the team considers it worth making.
Six Pitfalls and What Actually Worked
Six failure modes repeat in early programs. Punishing non-compliance before the process is mature fights the wrong battle. Over-optimizing for coverage buries the reviews that matter most. Satisfying developers too well before adding congestion control floods the queue. Waiting for leadership permission over-designs the first iteration before getting any real feedback. Reacting to a single bad review as if it reflects a pattern erodes a healthy program. And changing the process without measuring side effects introduces new problems while solving old ones.
Q&A
What was the biggest challenge using LLMs for threat modeling? LLMs produced too many generic risks rather than findings specific to the organization’s architecture; filling the context window with design docs and feature documentation improved result quality. ▶ Watch (29:55)
Did survey length affect response rates? The team targeted under 5 minutes, stated the estimate in the ticket header, used yes-or-no questions by default, and added conditional branching so UI-only features could exit early without answering everything. ▶ Watch (33:24)
Did teams resist engaging with security review? Rarely, as long as they understood the process had a clear endpoint and would not block their release date. ▶ Watch (34:49)
Notable Quotes
the satisfaction of our program has a net promoter score of 52 Felix Matenaar · ▶ Watch (3:09)
you you just end up in the wrong battle Felix Matenaar · ▶ Watch (19:09)
becoming a victim of your own success Felix Matenaar · ▶ Watch (20:33)
Key Takeaways
- Earning NPS 52 required three iterations; starting with full coverage revealed how to filter it down.
- Tech leads in a feature area have more triage context than a central security team reviewing everything.
- Collecting data before changing a process is what separates a real pattern from an outlier complaint.
About the Speaker(s)
Felix Matenaar is Head of Product Security at Asana. He has 12 years of professional security experience and a prior decade of education in building and breaking at hacker spaces. His work spans exploit generation and automation.