AI
Two production problems: security routing and AI-driven threat modeling
Stripe’s security team faced two concrete bottlenecks: too many security reviews for too few engineers, and a flood of new AI-assisted development projects demanding security guidance. The first bottleneck is a routing problem — with multiple specialized security teams, getting a developer to the right one without wasting engineering time. The second is a threat modeling problem — volume has outpaced human capacity. Hallucinations are the shared failure mode. Route someone to the wrong team and you’ve recreated the exact delay you were trying to eliminate.
Threat modeling agent architecture: modular, sequential, and tool-intentional
The threat modeling agent runs three layers in sequence: input agents pull context from Google Docs and Slack threads, specialized security agents analyze in parallel, then output agents format results. Each specialized agent covers one review category and must hit a baseline of required questions covering data sensitivity, transport protocols, and auth. Company-specific guidance tools feed those agents, because generic LLM advice isn’t actionable at Stripe’s risk level.
Sequential ordering was deliberate. Open-ended orchestration caused the orchestrator to skip relevant agents entirely.
“when we gave our orchestrator agent too much agency, it wouldn’t always run the relevant specialized agent that we wanted it to” — Jeffrey Zhang
Security routing agent: from one-step prompt to minimal agentic tool set
The security routing agent started as a single LLM call with no tools, just a prompt stuffed with descriptions of every security team at Stripe. It was fast. It also hallucinated whenever a user’s question touched internal tooling or terminology the model had never seen. Giving the agent access to research tools fixed accuracy but created a new problem: the agent took around 10 minutes to run. Iteratively removing tools one-by-one while re-checking accuracy got the routing agent from ~10 minutes to ~30 seconds with just two tools. Each removal was tested against a baseline question set, and a tool stayed out only if accuracy held. Two tools survived.
Evaluation pipelines: LLM-as-judge with human golden sets for threat modeling
Deterministic matching failed first. The agent identified the correct risk but assigned it different category labels across runs. Keyword matching had the same problem. Stripe turned to LLM-as-judge, but with a safeguard: security engineers authored golden test cases from past reviews, and the LLM scored semantic equivalence against actual agent output. Adding authorization and SSO as mandatory domains raised accuracy 10%. Benchmarking models on duplicated test cases added another 10%. The pipeline’s clearest proof: a JSON formatting instruction dropped accuracy 10% because the agent focused on output structure instead of security reasoning — a regression invisible without systematic scoring.
“threat modeling is often more of an art than a science” — Jeffrey Zhang
Phased rollout, human-in-the-loop thresholds, and output format strategy
Stripe started by targeting a narrow subcategory of security reviews with consistent risk patterns, ran the threat modeling agent in shadow mode, and iterated until accuracy hit 80%. That threshold was deliberate: with a human confirming threats before they reach engineering teams, a perfect score isn’t required. The architecture held a lower bar than it would if the agent sent findings directly.
Different audiences then drove the output format split. Security engineers wanted a summary. The internal threat tracking tool needed structured data for metrics. For incomplete tickets, the agent passes its work to a conversational agent rather than guessing. When information is missing, it flags the gap explicitly — marking status as unknown — instead of filling it in.
Key learnings: AlphaEvolve limits, eval-first discipline, and garbage-in handling
AlphaEvolve — Google DeepMind’s evolutionary prompt optimizer — failed on open-ended language. Trials added two words or paraphrased the whole prompt without moving the eval score (14:27). It works when permutations are finite; natural language is not.
“humans in the loop aren’t optional” — Siddh Shah
The eval pipeline caught regressions single runs missed — a JSON formatting addition dropped accuracy 10%. Build it early. On vague tickets, the agent hallucinated encryption and rotation details not in the ticket. The fix: teach it to say “I don’t know” rather than fill gaps.
Q&A
Was accuracy the only metric used in evals, or did you try other metrics as well? Offline accuracy against the golden test set was the only metric so far, with online evaluation — using user approve/deny actions in the threat modeling tool as a feedback loop — planned for the next iteration. ▶ 17:30
Have you looked into leveraging the Maestro threat modeling framework for AI scenarios? No adoption yet — the speaker acknowledged Maestro could be applicable depending on the category of security reviews but said they’d been building on their existing MITRE-based processes. ▶ 18:57
Notable Quotes
threat modeling is often more of an art than a science Jeffrey Zhang · ▶ 6:56
humans in the loop aren’t optional Siddh Shah · ▶ 15:30
garbage in input really means garbage output Siddh Shah · ▶ 16:34
when we gave our orchestrator agent too much agency, it wouldn’t always run the relevant specialized agent that we wanted it to Jeffrey Zhang · ▶ 4:35
Key Takeaways
- Build a human-curated golden test set before tuning prompts — eval pipelines catch regressions that look fine in isolation.
- Match agent architecture to task structure: sequential orchestration for threat modeling, minimal agentic tool call for open-ended routing.
- Teach agents to flag missing information instead of hallucinating; ‘I don’t know’ is a feature, not a failure.