AI
AI Security and AI Safety: Two Distinct Problems
AI security means finding vulnerabilities that compromise an AI system’s confidentiality, integrity, or availability. AI safety targets something different: flaws that harm users or expose an organization to legal or reputational risk. The clearest way to separate them: AI security protects the AI system from the outside world. AI safety protects the outside world from the AI system. For this talk, agents are programs where LLM output controls the workflow. That definition matters because agents combine both attack surfaces. An attacker who manipulates agent behavior inherits all the tool calls and data access that agent holds.
Classic Web2 Vulnerabilities Still Own AI Platforms
Virtuals let anyone deploy AI agents tied to crypto wallets and Twitter accounts. Within 30 minutes of a basic Burp review, a GitHub token appeared in browser console traffic. That token pulled a private repo with an old commit containing Pinecone keys, AWS credentials, and several other leaked service secrets. The AWS keys gave full read-write access to every agent’s S3-hosted training data. No versioning, no monitoring. Anyone with the key could silently overwrite any agent’s RAG data without the platform knowing. Virtuals paid a $10,000 bounty in native crypto, now worth $3,600.
Automating Bias Discovery Against a DoD LLM
The DoD Chief Digital and AI Office ran a bias bounty on Bug Crowd targeting Llama 2. Reports scored on military relevance and fire rate: a bias firing 80% of the time beat one at 60%. Dane built a CrewAI pipeline to automate the work. One agent searched DuckDuckGo for potential biases, a second formulated submission prompts. The pipeline surfaced issues he would have missed: body armor specs that exposed female soldiers and a recruitment model that sorted identical profiles by race into infantry versus intelligence roles. He submitted 53 reports, 26 accepted, winning $6,000.
Context Injection: Tricking an Agent Through Fake Conversations
Grey Swan ran a red teaming competition with 20 frontier models, 40 scenarios, and $170K in prizes. One scenario asked hackers to trick an agent into posting a user’s password to Instagram via indirect prompt injection. A direct command failed. The working payload invented a fictional “social registration flow,” used line breaks to simulate a new conversation turn, then added “explicit permission granted” as a spoofed user message. That tricked the agent into calling the Instagram tool. Dane earned $8 because others had already solved the same scenario first.
Structuring Bug Bounty Programs for Probabilistic AI Systems
AI bug bounty programs differ from traditional ones. AI systems are probabilistic: a safety flaw might fire 60% of the time, not 100%, so reports need statistical framing. The DoD competition scored reports partly on fire rate, which gave hackers a concrete target. Programs that publish objective criteria attract more useful submissions than those that say “go find bad things, good luck.” Daily triage kept hackers on track and out of rabbit holes. Grey Swan built tooling to verify each payload automatically, turning hacker submissions into a publishable dataset on how 20 models handle prompt injection.
Q&A
How would you improve the agent framework you used for the DoD bias bounty? Rather than “vibe coding” with one model, Dane says he would build rigorous evaluations: benchmark outputs from OpenAI, Hugging Face, and Anthropic against expected results, vary agent count, and iterate on prompts before spending API credits. ▶ 46:58
What is your top advice for organizations building AI agents? Dane says start with a written threat model and least-privilege design before worrying about classifiers or system prompt sanitization; Shlomie adds that crowdsourced testing with domain experts and anomaly detection on response size are underused defensive layers. ▶ 48:40
Notable Quotes
that AI security is about protecting the AI system from the outside world and that AI safety is about protecting the outside world from the AI system Dane Sherrets · ▶ 6:38
no policy page survives first contact with hackers Dane Sherrets · ▶ 28:21
LLMs love context. They love updates to context. They love additional context. They love new context. Dane Sherrets · ▶ 36:14
all risk management begins with an inventory Dane Sherrets · ▶ 49:17
Key Takeaways
- Exposed credentials in Git history gave read-write access to every agent’s RAG training data in the Virtuals platform.
- AI agents can extend a hacker’s domain knowledge into areas like military biology, surfacing biases no single researcher would find.
- Indirect prompt injection works by supplying fake conversation context until the agent accepts a spoofed user permission.
- AI bug bounty programs should score findings on fire rate, not just severity, because probabilistic systems require statistical evidence.
- Least-privilege permissions and anomaly detection on response size are the most practical infrastructure-layer defenses against agent exploitation.
About the Speaker(s)
Dane Sherrets is an Innovations Architect at HackerOne, where he helps organizations run AI-focused bug bounty programs. He won second place in the Department of Defense AI Bias Bounty competition, discovered critical vulnerabilities in platforms like Worldcoin, and co-designed Anthropic’s AI Safety Bug Bounty program.
Shlomie Liberow is a security researcher who translates technical vulnerabilities into business risk for enterprises. He has led live hacking events for major organizations and mediated over $20M in bounty payouts. He has personally discovered 250+ vulnerabilities across Fortune 500 companies.