From Fixed Code to Stochastic Behavior

▶ Watch (4:27)

Traditional pen testing is binary: a SQL injection either works or it doesn’t. Generative AI replaces that with behavior, not code. Models are stochastic. Same input, different output even with temperature zero. Red teamers manipulate persuasion, not exploits. Finding a vulnerability means observing inconsistent responses, not system crashes. The repeatability problem means a developer may not reproduce the attack, making validation impossible.

The Lie Machine: Why Black Box Testing Is Useless

▶ Watch (9:50)

LLMs are confident liars. Hallucinations are an inherent flaw in the stack. A chatbot that appears to perform SQL injection is actually hallucinating the entire database. The only way to know if a system prompt leak is real is to compare it against the actual system prompt. Black box red teaming is useless because it cannot distinguish a real finding from a fabricated one. Without data access, red teamers are throwing money away.

Jailbreaks vs. Prompt Injection: Two Attack Surfaces

▶ Watch (21:03)

Jailbreaking attacks the model’s built-in safety training. Prompt injection attacks the framework: system prompts and guardrails added by the consumer. Direct injection impacts only the user; indirect injection poisons data sources like resumes or logs. Context poisoning uses contaminated data in RAG or web searches. An attacker can plant prompt injection in a Wikipedia page and later trigger it through the model.

Crop, Crescendo, and Adversarial Suffixes: Real Attack Patterns

▶ Watch (29:09)

Crop attacks bypass guardrails by using cultural references. Asking for “Mickey Mouse smoking” becomes a request for “the classic rodent cartoon character who lives in the place where Super Bowl victors go to celebrate doing the thing that made restaurants smelly in the 50s.” Crescendo attacks start with a fairy tale and escalate to graphic descriptions of children in cages by slowly building context. Adversarial suffixes use mathematically related words to trigger the same neurons as a forbidden request.

Agentic Systems and the Inherent Flaw

▶ Watch (34:51)

Agents take real action, making them dangerous. The machine lies and takes actions based on lies. Multi-agent systems spiral when one goes off the rails and the other follows. Prompt injection is an inherent flaw in transformer technology. There is no hierarchy between trusted and untrusted data once in the context window. GPT-5 is the first model trained to respect tool output hierarchy, but the problem persists. Security must focus on blast radius, not elimination.

Notable Quotes

This whole thing is hallucinated, including the initial results. Jason Ross · ▶ Watch (11:18)

Black box red teaming for AI isn’t a thing you can do. It’s useless. Jason Ross · ▶ Watch (15:42)

You can’t sanitize language. It’s too big. Jason Ross · ▶ Watch (38:37)

We have no way to diffuse the bomb. It It’s there and it’s going to go off. Jason Ross · ▶ Watch (57:29)

Key Takeaways

  • Black box red teaming without data validation is useless for AI.
  • Prompt injection is an inherent flaw; accept it and control blast radius.
  • Tools like Crop Duster and Tapper automate attacks but require customization for agents.

About the Speaker(s)

Jason Ross is a product security principal at Salesforce who has spent the last four years breaking AI. He leads the OWASP GenAI Red Team Initiative and co-leads the OWASP Generative AI Security Project core team. He is also a DEF CON NFO goon.