AI
Why SAST Coverage Is a Scaling Problem
SAST tools must understand source code automatically. Every language, library, and framework compounds complexity. Benchmarks like OWASP Java test specific syntactic patterns, not real enterprise code. A library like Django has 9,000 callable functions. Manual rule writing for all of them takes months. Most tools cover only the top 100 or 200 libraries, but customer code uses thousands more. The gap is not theoretical.
The Long Tail: 84% of Repositories Uncovered by Top 1,000 Libraries
Merloni examined 80,000 Python, Java, and JavaScript repositories. If a SAST tool perfectly covered the top 1,000 most-dependent libraries, only 16% of repos would have complete coverage. Even covering the top 20,000 libraries covers only 30-46% depending on ecosystem. Python’s PyPI alone had 7,000 distinct dependencies across those repos. Dependency overlap between any two repos averages 10%. Every repo uses a different set of libraries.
AI Pipeline: From Library Prioritization to Annotations
The pipeline starts with static analysis to extract function signatures, arguments, and documentation. AI agents then classify each function: is it a data source or sink? Which vulnerability type does it relate to? A security review agent cross-checks the classifications. The human researcher gets a single web interface showing each annotation with context. They thumb-up or thumb-down. Accepted annotations become stubs — function bodies replaced with type-safe tags like semgrep.sink.sql. A compiler turns stubs into SAST rules, guaranteeing consistency and preventing typos.
Rule Synthesis and Results: 700 to 7,000 Rules in Two Weeks
The compiler generates rules automatically. It merges similar patterns (e.g., three database-execute methods into one rule). If Semgrep improves its engine, the same compiler output becomes better rules without rewriting anything. In two weeks, one person 10x’d Python rule count from 700 to 7,000. The number of supported libraries doubled in the same period. Previous support for that many libraries took six months. The new rules find real vulnerabilities — proving the long tail holds exploitable bugs.
Q&A
How do you know when coverage is complete? You cannot stop; new libraries appear daily, so automation must continue. ▶ 43:59
How do you validate the AI-generated annotations are correct? Human-in-the-loop with thumb-up/thumb-down feedback; rules are then run on thousands of repos to catch false positives. ▶ 45:20
If you secure the core framework used by long-tail AI apps, can you claim complete coverage? No — each company uses a specific subset of libraries; even one obscure dangerous function left unanalyzed can be a problem. ▶ 47:21
Notable Quotes
“if we had rules … for all of the top 1,000 most dependent upon libraries we would cover 16% of the repositories” Claudio Merloni · ▶ 18:45
“in a couple of weeks basically one person was able to 10x the number of rules we have for Python literally going from something like 700 to 7,000” Claudio Merloni · ▶ 39:14
“the more rules you produce the more it gets hard if you want to use them so you need a system that can also scale with the amount of content you’re going to produce” Claudio Merloni · ▶ 41:47
“there is going to be a repository where not having that one rule it means missing one vulnerability” Claudio Merloni · ▶ 40:13
“I don’t think you can be done because the libraries you’re going to kind of … what you’re looking at today is not going to be what you’re looking at tomorrow” Claudio Merloni · ▶ 43:59
Key Takeaways
- Top 1,000 libraries cover only 16% of real-world repositories.
- Dependency overlap between repos averages 10%, requiring broad coverage.
- An AI pipeline with human review can generate 10x more SAST rules in two weeks.