AI
The First Try Fallacy: Why AI Testing Breaks Reproducible Security
LLMs are non-deterministic. The same input produces different outputs. That breaks the standard security testing workflow. With SQL injection or XSS, one proof-of-concept payload works every time. With AI, Haddix said he runs each test case between 5 and 15 times to confirm a true positive. This changes how teams report bugs. A customer’s developer might try a leaked system prompt once, see it fail, and call the finding invalid. The fix requires a new tester habit: repeat each attack string, monitor response variance, and treat a single refusal as inconclusive.
Methodology for Holistic AI System Attacks
Haddix built a 6-step methodology because no existing framework covered AI workflows. Step one: identify every input vector for text into the system. Step two: attack the ecosystem of web apps that front the AI. Step three: attack the front-end model itself. Step four: leak and subvert the system prompt for every model in the chain. Step five: attack data stores like RAG databases. Step six: pivot from the AI system to other internal systems. He said most real attacks combine steps. Pure jailbreaking of the front-end model rarely shows business impact without a pivot to data or other users.
Blind XSS Through a Healthcare RAG Pipeline
A healthcare customer built a multi-agent system that ingested PDFs, extracted content, ran business analysis, and cataloged results. Haddix’s team identified the inputs: a file upload portal and an S3 bucket. They crafted malicious files with JavaScript in metadata, binary data, and even QR codes. The first AI in the pipeline passed the files through. Months later, a human reviewer opened a dashboard. The JavaScript executed a fake Microsoft login prompt. The team captured the cookie and logged into the victim’s Microsoft tenant. The files were also ingested as training data, causing callbacks for months after the test.
API Key in the System Prompt: An Automotive AI Failure
An automotive company built an internal search tool over three data sources: a QA knowledge base, a specification database, and domain notes. The tool used open-source models with an agentic architecture. Early AI engineering practices meant the system prompt had to include full API endpoints and keys for tool calls. Haddix leaked the system prompt from one agent. Inside it was an API key for Jira and Confluence. The team stole the key and gained direct access to those internal systems. They also used prompt injection to extract full patent acquisition costs and fault tolerances the RAG database was supposed to restrict.
SIM Vendor: Poisoning the AI’s External Data Source
A SIM vendor integrated an AI sidebar that enriched IOCs with data from a third-party API. Haddix told the AI to use a new enrichment URL. The AI fetched code from the attacker’s server. That enabled cross-tenant attacks. The team stole SIM telemetry from other tenants, poisoned the data set, and made the AI mask real attacks in analyst queries. Haddix called this a “newer test” and said it demonstrated how AI agents with tool-calling abilities create supply-chain attack surfaces that traditional web app testing misses.
Manual Skill Still Beats Automated Scanners by 80%
The Hacker Prompt V2 CTF tested defenses against 10 automated scanners and human attackers. Automation bypassed only 20% of the protections. Humans bypassed every single layered defense. Haddix said automated frameworks are not granular enough for security standards. He also cited Sam Altman’s admission that prompt injection will never be fully trained out of models. The reason: transformers are text predictors. As long as you can fake text, you can trick the model. Haddix urged testers to follow both academic papers and the jailbreak community to learn tricks.
Notable Quotes
So, the first one we looked at was Pliny the Prompter’s Liberatus repo. Jason Haddix · ▶ Watch (29:05)
And he’s like, “I was wrong.” He’s like, “There’s always going to be about a 2% of which prompt injection will still be available against modern models just cuz how they’re made.” Jason Haddix · ▶ Watch (53:07)
These models are just text predictors. Jason Haddix · ▶ Watch (53:31)
The automation was able to only bypass the protection 20% of the time. The humans who manually attacked these systems were able to bypass every protection and layered version of protections. Jason Haddix · ▶ Watch (54:56)
Key Takeaways
- Test each prompt injection payload 5 to 15 times to confirm it works.
- Attack the whole AI supply chain, not just the front-end model.
- Leaked API keys in system prompts remain a common critical finding.
- Automated scanners still miss 80% of attacks that humans find today.
About the Speaker(s)
Jason Haddix is the CEO and “Hacker in Charge” at Arcanum Information Security. He has 20 years in offensive security, previously serving as CISO of Buddobot, CISO of Ubisoft, and Head of Trust at Bugcrowd. He is ranked 57th all-time on Bugcrowd’s leaderboard and has spoken at DEFCON, BlackHat, and RSA.