AI
Autonomous Pentest Architecture
The system has three main components. A coordinator acts as a pentest manager. It handles analysis, priorities, recon, and task assignment. Specialized agents execute tasks and look for vulnerabilities. Validators act as a second pair of eyes. They remove false positives and hallucinations from the AI. A proxy sets boundaries to keep agents in scope. Agents access an attacking machine with built-in tools adapted for a headless browser. The coordinator also tracks coverage and uses any provided documentation, like Swagger files.
Validator System and False Positive Elimination
Validators are per vulnerability type. For XSS, a headless browser receives the link with the payload and checks that a pop-up or console message appears. The validators are strict. They reject false positives. Some vulnerability classes achieve a zero false positive rate. The validators also catch AI cheating. Models often try to please the user by fabricating results. The validator asks how the agent achieved the finding. If the agent cannot prove it, the finding is discarded.
Alloy Models and Model Selection
XBOW introduced the alloy models concept. Instead of using the same model for every query, the system flips a coin and sends the query to one of two models. The coordinator saves and transfers context between models. Every combination of two models outperformed single models. The more different the models, the better the performance. Some models are better at specific tasks, like XSS or crawling. For a focused task, using a single strong model may be better than mixing.
Safety Boundaries and Policy Enforcement
The system does not perform post-exploitation. Agents stop after finding a vulnerability. A policy checker blocks destructive actions like deleting database records. The proxy ensures agents stay within the defined scope. For bug bounty programs, human oversight is required for report submission. The AI generates the report after validation, but a human submits it to HackerOne. This prevents the AI from accessing the HackerOne account directly.
Q&A
Is there a human in the loop? Human oversight is only for report submission. The AI does everything else, including validation. ▶ 3:43
What vulnerability types are supported? RCE, file read (XXE, path traversal), SQLi (blind, time-based), XSS (stored, reflected, DOM, blind), open redirect, SSRF, SSTI, cache poisoning, IDOR, and exposed secrets. ▶ 6:08
Do you limit the AI’s steps? No, agents have complete freedom. The coordinator assigns tasks like “go for XSS” and lets the agent run. ▶ 8:43
How do you validate credentials found in the application? That is part of the validator system. The team has ways to verify credentials without sharing details. ▶ 12:49
Are you still using CTF flags for validation? No. CTF flags were used in early benchmarks but not for live bug bounty. Flags now are just tokens to reward the model for passing validation. ▶ 19:07
Notable Quotes
the more different are those models uh the better they were performing Diego Jurado · ▶ 14:28
we don’t do post exploitation Joel Noguera · ▶ 11:54
we have a policy checker that verifies that um whenever we the model is trying to do something that might be uh um bad for the system avoids like performing that action Joel Noguera · ▶ 20:57
Key Takeaways
- XBOW’s autonomous pentest AI uses a coordinator, agents, and validators to find bugs.
- Alloy models (flipping between two LLMs) outperform single models in vulnerability discovery.
- Validators achieve near-zero false positive rates by verifying each finding programmatically.
About the Speaker(s)
Diego Jurado is a security researcher at XBOW. He has a background in bug bounty, penetration testing, and red teaming. He held positions at Microsoft Xbox, Activision Blizzard King, and Telefónica. He is in the top 38 all time on HackerOne and part of Team Spain, winner of the HackerOne Ambassadors World Cup 2023.
Joel Noguera is a security researcher at XBOW. He has nine years of experience in exploit development, reverse engineering, and security research. He has been in bug bounty since 2016, reaching the top 60 on HackerOne. He previously worked at Immunity Inc. and has presented at Recon, BlackHat Europe, EkoParty, and BSides.