AI
SAST Tools Miss 87% of Real Vulnerabilities
A 2023 research study comparing SAST tools against actual Java CVEs found they miss 87% of real-world vulnerabilities. After 20 years of evolution, static analysis tools still fail to detect almost nine out of ten real bugs in production code. Vendors claim full CWE coverage in sales pitches, but no vendor admitted gaps when buyers asked directly. Benchmarking exists to cut through that noise, to determine which tools find real vulnerabilities and which just generate alerts.
Synthetic Test Cases Hide the Real Failure Mode
The OWASP Benchmark has nearly 3,000 runnable Java test cases across 11 vulnerability categories. Best SAST tools score 83% accuracy on it. The problem: synthetic tests put sources and sinks one or two hops apart. Production applications route tainted data through dependency injection frameworks, service layers, and object boundaries. OWASP has no test cases for a SQL query reachable only through a specific Spring controller route, or a background job deserializing untrusted data from a queue. High scores on synthetic tests create a false sense of tool capability.
Real CVE Benchmarks Reveal an 87% Miss Rate
The Java CVE benchmark tested SAST tools against 165 real open-source Java CVEs. The best single tool missed 87% of them. Combining the four top SAST tools lifted detection to only 27%, but added two to three times more false positives. On patched code, false positive rates ran as high as 96%. Path traversal detection fell from 100% on synthetic tests to 16% on real CVEs. Deserialization CVEs went 91% undetected. Of 38 distinct Jackson deserialization CVEs, every tool detected zero.
Three Root Causes of SAST Blind Spots
Giannakidis identified three failure patterns. First, lack of runtime context: static scanners cannot observe which authentication framework is active, or how Spring and ASP.NET wire security declaratively through annotations at runtime. When a scanner cannot prove a code path is safe, it flags it, producing false positives or missing context-dependent paths entirely. Second, no standard sanitizer API: teams build custom encoders and validators; no scanner covers every permutation. Third, detecting the absence of a security mechanism requires whole-program semantic reasoning that no static tool achieves.
Why Runtime Is the Only Source of Truth
Application code accounts for less than 20% of what runs on the JVM or CLR. The other 80% is application servers, third-party libraries, and frameworks. Static analysis scans each layer in isolation. Many vulnerabilities only appear when the full stack runs together. The runtime is the only place where configuration effects, dependency wiring, and actual data flows are visible. RASP-based detection evaluates every payload inside its real execution context, making findings verified attacks rather than hypothetical paths. Current benchmarks have no test cases for any runtime-dependent vulnerability scenario.
Q&A
Does RASP provide the runtime-context alternative that SAST lacks for detecting real vulnerabilities? Runtime security is the only known way to identify context at execution time, and RASP evaluates every payload within its full execution context, making it a verified attack rather than a hypothetical path. ▶ 43:59
Are some vulnerability classes fundamentally beyond taint analysis, suggesting type-based approaches like Trusted Types instead? Different problems require different solutions, and even IAST has blind spots when JVM or CLR execution crosses into native OS calls where instrumentation loses visibility. ▶ 45:32
Notable Quotes
Believe me, the results are shocking. Apostolos Giannakidis · ▶ 18:23
Zero. Zero. Exactly. Apostolos Giannakidis · ▶ 22:26
runtime security is the only known way Apostolos Giannakidis · ▶ 44:44
Key Takeaways
- SAST tools miss 87% of real CVEs; benchmark scores on synthetic tests mislead buyers about actual capability.
- Combining four top SAST tools raised detection to only 27%, adding more false positives than real findings.
- Runtime is the only place where full-stack data flows are visible; current benchmarks ignore this entirely.
- Next-generation benchmarks need real CVE test cases, runtime-context scenarios, and severity accuracy evaluation.
About the Speaker(s)
Apostolos Giannakidis leads the Threat Modeling program at Microsoft Identity. Before joining Microsoft, he served as VP of Application Security at JP Morgan Chase and led security strategy at Waratek. He has been acknowledged by Oracle.