AI
The First-Try Fallacy in AI Pentesting
LLMs are non-deterministic. The same prompt sent twice returns different answers. For pentesters, this breaks a basic assumption: that a working payload keeps working. Haddix’s team leaked a customer’s complete system prompt at a 60% hit rate. When they handed the attack string to the client, it failed to reproduce on their machine. The fix is to test each prompt injection 10 to 15 times before calling a finding confirmed. That volume makes AI pentesting slower than traditional web work.
Seven Steps From Input to Pivot
Arcanum’s AI assessment runs seven steps, starting with mapping all system inputs before touching the model. Most testers jump straight to jailbreaks. The methodology works outward from the attack surface first: every channel that carries data into the AI chain, from file uploads to direct API calls. From there, the team attacks the surrounding infrastructure, the model, the prompt engineering, the data layer, and the application before pivoting. The model is one piece, not the whole target.
Healthcare Documents as Injection Vectors
A healthcare client built a multimodal agentic system to ingest patient documents. The only input was a file upload and an S3 bucket. Haddix’s team embedded JavaScript and prompt injection in every layer of crafted PDFs: filenames, metadata, binary sections, and text over images. The payloads passed through the LLMs and executed in the human-review interface. The client also used processed documents for a second training round, so the team received callback pings for weeks after the engagement ended, forcing a revert to a stable model version.
Production AI Systems Ship Without Security Review
An automotive company’s LLM agent carried Jira and Confluence API keys in its system prompt. Leaking the prompt handed Arcanum direct access to both, plus a path to plant Evilginx phishing pages and steal session cookies. A tech company had shipped a Salesforce-connected sales chatbot through OpenAI without security review. Prompt injection gave full read/write access to customer records. On Amazon.com, a former NSA analyst found the Rufus shopping chatbot bypassed its safety tuning with simple encoding. Amazon makes Bedrock’s guardrails. The Rufus team forgot to enable them.
A Prompt Injection Taxonomy With 10 Trillion Combinations
Haddix built a prompt injection taxonomy modeled on Metasploit. Attacks break down by intent (what the attacker wants the AI to do), technique (how to induce it, such as the Russian doll method or narrative injection), and evasion (how to slip past guardrails using Morse code or Unicode hidden inside emojis). The taxonomy has just under 10 trillion combinations when mix-and-matched. The team also released Parcel Tongue, a self-hosted page with over 90 encoding transformations, covering formats from Klingon to binary.
Automated Red Teaming Tops Out at the 40th Percentile
Sander from learnprompting.com runs one of the largest annual AI red-teaming events. His October findings: humans beat every automated tool at creative prompt injection, regardless of whether the defense runs at the prompting, training, or filtering layer. The best automated vendors in the competition never broke the 40th percentile against human competitors. Open-source scanning tools can cover that baseline, but they leave the other 60% untouched. An organization that relies only on automated scanning will miss what a skilled human finds. Manual red teamers are not optional.
Notable Quotes
API call was both read and write. Jason Haddix · ▶ 28:12
It is enterprise level. It stymies me Jason Haddix · ▶ 30:31
how fast this is moving. Jason Haddix · ▶ 30:43
humans by far destroy machines Jason Haddix · ▶ 46:19
Key Takeaways
- Non-determinism means testing a prompt injection 10 to 15 times before calling it confirmed or ruled out.
- System prompts can hold live API keys; leaking one often hands access to connected services like Jira or Salesforce.
- Even the team that builds guardrails can forget to turn them on for their own product.
- Automated AI red teaming tools top out at the 40th percentile compared to skilled human attackers.
- Prompt injection bypasses work through encoded text, Unicode inside emojis, custom character mappings, and truncated instruction chains.
About the Speaker(s)
Jason Haddix, known online as jhaddix, is the CEO and “Hacker in Charge” at Arcanum Information Security and field CISO for flare.io. He brings 20 years of cybersecurity experience, with prior roles including CISO at Buddobot, CISO at Ubisoft, Head of Trust/Security/Operations at Bugcrowd, and Director of Penetration Testing at HP. Arcanum provides security assessment and training services.