Why auth and authz bugs resist classical automated detection

▶ Watch (00:06)

Auth bugs are hard to automate because there is no single auth standard. HTTP basic, OAuth 2.0, signed URLs, JWTs, OIDC, Kerberos, magic links, TOTP, LDAP—and then, as Dolan-Gavitt put it, the most common mechanism of all: custom auth, where the developer said,

“Yeah, I’m pretty sure I know how to do this.” — Brendan Dolan-Gavitt

No classical validator can cover that spread. Even when developers implement OAuth 2.0, it is not necessarily spec-compliant. The only way to tell whether an endpoint is properly protected is to verify behavior at runtime, which is where AI becomes necessary.

Building a two-checkpoint login validation system

▶ Watch (02:19)

XBOW gives an agent a browser and credentials, then asks it to log in. That alone isn’t enough. False positives are common, so the system runs two validators: one confirms the agent is actually logged in, the second checks that it has reverse-engineered the app well enough to translate browser state into raw API requests. Both checkpoints must pass. The trick, Olesen explained, is finding spots in the app where logged-in and logged-out states produce a visible contrast — because a single API response, stripped of context, rarely tells you whether the auth material mattered at all.

Turning login validators into authentication bypass detectors

▶ Watch (03:57)

The same validators built to confirm login become bypass detectors with one change: strip the password and rerun them. If the agent passes both checkpoints without credentials, that’s an authentication bypass, confirmed automatically. Strip MFA instead, same result. The Horizon 3 hardcoded-secret demo (04:21) shows how far this reaches: the agent reads source code, finds a hardcoded JWT secret, forges an admin session token, loads it into the browser, and the validators confirm admin access.

Authorization bypass via auth transmogrification

▶ Watch (05:40)

“So, this is where I feel like we go from evil to diabolical.” — Brendan Dolan-Gavitt

Authentication failures are rare. Authorization failures aren’t. Auth transmogrification handles the harder case: the agent writes a script that replays any admin request under a low-privilege session. Endpoints with differential responses become attack targets. The validator holds the admin response as ground truth; the attack agent, given only the URL, must reproduce it. Redmine endpoint prioritization via differential responses (10:34) surfaces /projects as the target. Redmine live trace: admin_projects=1 parameter bypass (12:08) shows the agent reading source code and, by iteration five, finding that admin_projects=1 exposes the private project. Validator confirms. Done.

Limitations: IDORs, script quality, and self-generating test suites

▶ Watch (14:35)

Classical IDORs fall outside this approach. When two users hit shopping_cart/5 and shopping_cart/7 and both see their own data, the differential-response signal disappears. Deciding whether that constitutes a bug requires judgment about the application—something the system deliberately avoids.

The second limit is script quality. Everything depends on the transmogrification script the agent writes, and mistakes there ripple through every subsequent test. The mitigation: let the agent browse the site with a browser, record every JavaScript-triggered request and response, and use that traffic as a built-in test suite that validates the script before it runs attacks.

Q&A

We focused on one specific validator—can you give a sense of how you came up with the approach and what other validator types you built? The auth bypass idea came from reusing login validators already built to keep XBOW reliably authenticated during pen tests—Vincent and Brendan implemented it at a hackathon, found real bugs, and the same validators stress-tested against internal banking apps gave them the robustness to repurpose for offensive detection. ▶ 16:05

Is this about enforcing proper authentication for AI agents in agentic workflows, or is it purely offensive security for web apps? Purely offensive: the system targets existing web apps to find auth and authz vulnerabilities before they ship, not to secure agents themselves. ▶ 17:19

Notable Quotes

Yeah, I’m pretty sure I know how to do this. Brendan Dolan-Gavitt · ▶ 1:47

So, this is where I feel like we go from evil to diabolical. Brendan Dolan-Gavitt · ▶ 5:40

it’s not fixed yet. So we’re operating under what I call the Bureau de vulnerability program where you buy us a beer and we’ll tell you about it. Brendan Dolan-Gavitt · ▶ 14:24

we also found that when we’re giving it these very, very targeted tasks, we don’t need very, very long-running agents to find these vulnerabilities usually. Brendan Dolan-Gavitt · ▶ 13:50

Key Takeaways

  • Build two validators—login confirmation and API-level session reuse—before trusting any agent is authenticated.
  • Strip credentials, re-run validators: authentication bypasses reveal themselves without needing to understand the auth mechanism.
  • Generate differential endpoint lists with auth transmogrification, then send a focused agent after each target rather than open-ended crawling.