The Manual Rule-Writing Bottleneck

▶ Watch (3:08)

Fifty-thousand supply chain vulnerabilities were estimated in 2025, up from 40,000 in 2024. 2024. Security researcher “Max” starts each day by checking a queue of incoming CVEs. For each one, he reads the description, finds the fixing commit, and studies the source code to understand how the package is used by real consumers. After 30 minutes of clicking through code and guessing, he writes one Semgrep rule. Then 20 more CVEs remain. That workflow does not scale as the CVE count grows.

Brat: A Deterministic Pipeline with Tasteful AI

▶ Watch (18:49)

Brat replaces that manual routine with three steps: research, test generation, and rule generation. The research step uses LLMs to scrape READMEs, API docs, and GitHub usage examples. It also builds a call graph from the patching commit. Functions changed by the fix are marked as vulnerable. A backward graph search marks all transitive callers as potentially vulnerable too. The graph search is deterministic — same input, same output. LLMs are confined to small subroutines like scraping and parsing.

Why Composability Beats One-Shot Agents

▶ Watch (33:00)

An agent asked to write a rule for 20 vulnerable functions often returns 19, drops one symbol, or produces a rule that does not parse. Brat splits the problem by function. For each public caller in the code graph, it generates a test snippet and a pattern independently. Each subpattern is verified against its test. Failures are isolated: rerun one subproblem, keep the nine that worked. The patterns are stitched into a single rule. The system is auditable at every step — the researcher sees the test, the pattern, and the pass/fail result.

Case Study: d-value Parse vs. Code Graph

▶ Watch (40:16)

A prototype pollution CVE in the npm library d-value. An agent was asked to research and write a rule. It correctly identified the vulnerability in d-value.parse because the unflatten helper did not block __proto__ keys. But the agent missed that unflatten is itself a public API function exposed by the library. The code graph showed hydrate (the real problem) called by unflatten called by parse. Both parse and unflatten were blue public nodes. The agent’s rule only flagged parse, leaving a gap. The deterministic graph search caught the missing node.

Results and Savings

▶ Watch (36:58)

Brat authored nearly 3,500 rules over two years. Researcher time for a ready-to-merge rule dropped from an average of one hour to about 10 minutes. “Very helpful” rules take 20 minutes; “helpful” rules take 45 minutes. The tool backfilled 2,000 Rust rules in two hours by running all CVEs in parallel. When 50 CVEs arrived in a single night, Brat handled them without human overload. The human-in-the-loop stays at the end, reviewing research summaries and verifying the generated artifacts.

Notable Quotes

Non-agentic code is deliberate code. When I write code that does something, some workflow, and I ground it in an actual script, I’m being very deliberate. Brandon Wu · ▶ 16:44

I believe that pure, simple, and verifiable parts make a stronger product. Brandon Wu · ▶ 21:07

We did not replace the human workflow. We just have a trusted mostly deterministic pipeline with a human in the loop. Brandon Wu · ▶ 39:26

The agent states the problem is in unflatten, but it didn’t realize that unflatten as a helper routine is also exposed by the library. Brandon Wu · ▶ 40:33

A tasteful amount of AI is the right amount. Brandon Wu · ▶ 43:14

Key Takeaways

  • Deterministic code graphs catch vulnerable public functions that agentic one-shot rules miss.
  • Splitting rule-writing into independent subproblems allows verification and rerun of only failed parts.
  • Brat cut researcher time from ~1 hour to ~10 minutes per rule across 3,500 rules.