AI
Prompt Injection: The Risk the Industry Is Pretending Doesn’t Exist
Most teams building AI products are scrupulously avoiding the topic. The AI hype cycle rewards speed, and prompt injection is an inconvenient risk — so companies pretend it doesn’t exist and leave users to bear the consequences. Even experienced AI developers and startup founders Bullen spoke with had only a vague understanding of the threat. Model-level defenses are getting better, but a 1% or even 0.1% failure rate is not acceptable in security.
“we basically have to assume prompt injections will happen and prevent compromised agents from causing harm when they do happen” — Andrew Bullen
The Lethal Trifecta and Lethal Bifecta: Two Frameworks for Agent Threat Modeling
Simon Willison’s lethal trifecta names the three conditions that let a prompt-injected agent exfiltrate data: untrusted content, private data, and egress. Bullen’s argument is that only one leg is cuttable. Removing untrusted content isn’t practical — feeding agents unstructured data is the point. Removing private data rules out most production agents; Stripe’s support tickets and emails can’t run on nothing. That leaves egress. For sensitive actions, the two-leg version — untrusted content plus an unguarded write path — he calls the lethal bifecta. An agent with a Google Docs tool can write private data to a public doc (07:06), bypassing every other control. The rule of thumb: production writes and broad outbound communications are sensitive actions; everything else probably isn’t.
Egress Lockdown and SaaS Proxying via Toolshed
Agents need data from the internet, not the ability to make arbitrary HTTP requests. Stripe’s answer is a safe search tool built on OpenAI’s API with external_web_access: false. Results come from cache. The agent’s data never leaves Stripe. SaaS is a harder problem: a compromised agent with a Google Docs tool can write private data to a public document and bypass every network-level egress block. Stripe closes this with Toolshed MCP proxy blocking non-Stripe SaaS tenant writes (10:15). Every third-party SaaS connection routes through a central MCP server, where a single rule prevents writes to any tenant that isn’t Stripe’s own. Google Docs, Figma, whatever. One proxy, one rule set.
Human-in-the-Loop Review Without Destroying Agent Usability
Human confirmation creates three distinct problems. Stopping the agent mid-task interrupts flow. Frequent low-stakes prompts cause review fatigue. And fatigue produces rubber stamping.
“with enough review fatigue, people are just going to mash confirm on any confirmation dialogues” — Andrew Bullen
Bullen’s fixes target each problem separately. Batch writes that don’t block forward progress so users review them later, not inline. For actions that can be undone, execute first and offer a revert. For the rest, route to an LLM pre-screener that flags genuinely suspicious calls before the human ever sees them.
Enforcing Guardrails at Scale: CI Checks, Service Tagging, and Tool Annotations
Stripe’s egress program predated AI agents, which made enforcement straightforward. Any service that talks to a foundation model gets tagged as an agent. From there, CI-time egress enforcement using Smokescreen service tagging (13:07) blocks deployment if the egress configuration doesn’t pass review. For tool-level guardrails, every tool in the agent framework or in the central MCP service requires annotations — properties like “production impacting write” or “broadcast data internally” — so the framework can decide automatically whether a human needs to approve it first. Agents increasingly skip defined tools and call APIs directly, so Stripe is now proxying traffic out of agent sandboxes and annotating API endpoints to apply the same sensitivity checks there.
Q&A
Is there value in aggregating human-in-the-loop review data at the team level to identify shared pain points? Bullen said they haven’t gotten there yet — the UX experimentation is very early — but the northstar is reducing how often agents stop while still applying human judgment at the right moments. ▶ 17:23
Do you still find value in threat modeling to partially stop prompt injections at design time? Yes, Bullen said detective and non-guarantee controls have a place, especially for customer-facing products, but Stripe leans on deterministic architectural controls because model-level defenses can’t be fully trusted. ▶ 18:18
Notable Quotes
we basically have to assume prompt injections will happen and prevent compromised agents from causing harm when they do happen Andrew Bullen · ▶ 3:44
security isn’t just about figuring out the secure way to do things. It’s about helping your organization figure out how to accomplish their goals in a secure way Andrew Bullen · ▶ 16:27
with enough review fatigue, people are just going to mash confirm on any confirmation dialogues Andrew Bullen · ▶ 11:30
Key Takeaways
- Assume prompt injection will succeed — architect agents so a compromised agent cannot cause real harm.
- Sever the egress leg of the lethal trifecta; block SaaS side-channels through a central MCP proxy.
- Make human-in-the-loop review tolerable with batching and reverts, or users will disable your guardrails.