AI
Signature-Based Scanners and Why They Failed at Scale
Isle started as a remediation company, not a scanner. The plan was to ingest existing scanner outputs and automate the fixes. To pick the best scanners, they ran 100,000 historical CVEs through commercial pattern-matching tools against the vulnerable code that was already known. What they found stopped them cold.
“the efficacy of those scanners were close to zero” — Ondrej Vlcek
Low single-digit detection rates. On bugs the scanners could have been trained on. That result pushed isle to build its own LLM-based scanner, and once it could reliably catch the known historical bugs, it started flagging new ones too.
Specific CVEs Found: OpenSSL, Browsers, and Traefik
Adam led with the most serious: a stack buffer overflow in an OpenSSL component used by email clients. An attacker controls field length in a data notation structure and overflows the stack. Existing mitigations limit remote code execution, but OpenSSL’s deployment breadth keeps the severity high. The pipeline also flagged a Traefik TLS bypass via inverted flag (08:07), where a Kubernetes config flag named proxy_ssl_verify sets insecure_skip to true, silently disabling backend TLS verification. A pattern-matching scanner reads the flag name and moves on. An LLM reads what the code actually does. isle also found one vulnerability each in Chromium, Firefox, and WebKit.
Responsible Disclosure and Avoiding AI Slop
Open-source maintainers are often volunteers operating on small budgets. Dumping raw AI-generated findings on them — even accurate ones — adds triage work they can’t absorb. isle learned this early. Every report they send includes root cause, a proposed fix, and full context. That discipline converted Daniel Stenberg, curl’s author and one of the loudest AI-slop critics, into an advocate. By last month he was at FOSDEM telling audiences the reports felt like magic.
The Multi-Stage Agentic Pipeline Architecture
The pipeline runs in two phases: a breadth-first pass that generates as many hypotheses as possible about what could be wrong, then a focused narrowing phase that double-checks each lead through agentic exploration, running code, and crafting proofs-of-concept. Multiple models assess each other’s conclusions from different angles, building consensus the way a security team would.
“we use a kind of uh multi-stage progressive refinement agentic pipeline where each stage acts as a sift to let through only like real CV grade findings that could uh cause real damage” — Ondrej Vlcek
Three things compound to keep false positives low: heavy parallelism (“Humans are mostly singlethreaded. LLMs or AI agents don’t suffer from this limitation,” Krivka noted), careful context construction that feeds each phase a tighter prompt, and a human checkpoint before any finding ships.
Deployment: PR Review Bots and the Race Against Attackers
The same pipeline now runs on every incoming PR for OpenSSL, OpenClaw, OpenEMR, and Apache, catching new vulnerabilities before they reach production. AI-generated code has accelerated the volume of changes hitting these repos, so catching bugs at the PR stage, not after the merge, is where the leverage is. Matt Caswell, CEO of the OpenSSL Foundation, called the catches impressive.
The offensive side has the same tools. Nation-state actors have near-unlimited token budgets and no disclosure obligations.
“we only prevent that what was called the vulner apocalypse if we move with urgency and move very fast to really preempt the bad guys from exploiting those vulnerabilities”
— Ondrej Vlcek
Q&A
Were the 500 CVEs previously known or all new? All 500 are new findings; the 130 minted CVE IDs are a trailing indicator because CVE issuance can take months and the backlog currently holds roughly 30,000 pending entries. ▶ 21:14
Should isle become a CNA, and does CVSS scoring need to evolve for AI-scale vulnerability discovery? Isle is actively working toward CNA status and expects to become one shortly; on CVSS, Ondrej called it an ‘explosive topic,’ said CVSS 4 is an improvement because it incorporates deployment context, acknowledged isle has ideas to augment the scoring system, but stopped short of committing to a specific alternative, saying they would rather collaborate with the community than build something unilaterally. ▶ 22:14
Notable Quotes
not only that our LM based approach could now find vulnerabilities that we were looking for that is those historical ones the kind of gold ones as we call them but he actually started identifying new vulnerabilities as well Ondrej Vlcek · ▶ 5:58
the efficacy of those scanners were close to zero Ondrej Vlcek · ▶ 5:33
we use a kind of uh multi-stage progressive refinement agentic pipeline where each stage acts as a sift to let through only like real CV grade findings that could uh cause real damage Ondrej Vlcek · ▶ 12:10
Humans are mostly singlethreaded. LLMs or AI agents don’t suffer from this limitation. Adam Krivka · ▶ 14:12
we only prevent that what was called the vulner apocalypse uh multiple times today if we move with urgency and move very fast to really preempt the bad guys from exploiting those vulnerabilities Ondrej Vlcek · ▶ 20:40
Key Takeaways
- Signature-based scanners hit near-zero detection on historical CVEs — stop treating them as a meaningful security control.
- Run your AI vulnerability pipeline as breadth-first hypothesis generation followed by multi-model narrowing, not a single-pass scan.
- Ship findings with root cause, proposed fix, and full context; raw AI-generated reports burden open-source maintainers and get ignored.