AI
Hacking as a Measurement Tool
Roldan opens with a reframe: penetration testing is a measurement tool, not a checkbox service. If the goal is to know how secure a system is, the output of any scan or test must be evaluated the same way you’d evaluate any measurement instrument. Accuracy matters. That means asking two questions: does it report everything real, and does it report things that aren’t real? The industry names for these failure modes are false negatives (omissions) and false positives (lies). Both degrade the measurement.
Why CVSS Severity Scoring Distorts Risk
CVSS assigns numerical severity scores to vulnerabilities, and most organizations use those scores to prioritize remediation. The problem: CVSS scores are linear, but risk is not. A severity-10 vulnerability is not twice as dangerous as a severity-5. Two severity-5 vulnerabilities don’t equal one severity-10. CVSS also groups everything above 9.0 into “critical,” implying a severity-9.9 is only marginally worse than a severity-9.0. Roldan argues both assumptions lead to bad remediation decisions.
CVSSF: Exponential Risk Scoring
Fluid Attacks built CVSSF by using the CVSS base score as an exponent. The result is exponential separation between high and low scores. Under their formula, a severity-4 vulnerability equals 16 severity-2 vulnerabilities, or 64 severity-1 vulnerabilities. A single severity-10 equals more than 262,000 severity-1 vulnerabilities. The proportionality constant P (set to 4 by Fluid Attacks) lets organizations tune equivalences. With CVSSF, risk exposure values are additive, letting teams track total exposure across a system and chart how remediation changes that number over time.
Experiment Design and the Vulnerability Baseline
Fluid Attacks ran a 10-month experiment on a single vulnerable-by-design Node.js application with MVC architecture, a single-page frontend, and a REST API. A senior hacker spent roughly two months establishing the ground truth: 1,221 vulnerabilities across 105 categories, totaling 4,461,499 CVSSF risk exposure. That baseline is the universe against which all tools and manual testers were measured. Over 73% of those vulnerabilities required authentication to reach, and those account for 90% of the total risk exposure. Most automated scanners tested couldn’t authenticate, limiting their access to the application’s highest-risk attack surface.
Manual Pentesting vs. Automated Scanners
A separate manual pentesting team found 872 true positives with zero false positives, measuring 420,000 CVSSF of risk exposure out of the 4,461,499-point baseline. That is an F1 score of 84%. The best automated scanner across all tools tested reached an F1 of 36.9%, below 50%, the point where a tool performs worse than random guessing. On risk exposure, the best tool detected less than 10% of what manual pentesting found. Roldan presented results without naming individual tools, comparing tools as a class against expert humans.
Why Automated Tools Can’t Close the Accuracy Gap
74.8% of the vulnerabilities in the target application were found only by manual pentesting. No automated tool detected them. The 49 days required is the tradeoff: tools run in minutes, expert hackers need weeks. The F0.5 score for manual pentesting was 93%, reflecting near-zero false positives. That score matters most to developers choosing a tool, since fixing false positives wastes sprint capacity. The conclusion: automated scanners help but miss the bulk of exploitable vulnerabilities and almost all high-severity risk exposure.
Q&A
Were any AI-based automated testing tools included in the experiment? No AI tools were included in the initial design, but Roldan raised the same question with his team and said AI tools may be added as the experiment continues. ▶ Watch (36:48)
Is the benchmark application open source, and how was the vulnerability baseline established? The app is a commonly available open-source vulnerable-by-default project; Roldan withheld the name. A senior hacker spent roughly two months doing manual pentesting to find the baseline of over 1,000 vulnerabilities. ▶ Watch (37:29)
How long did manual pentesting take compared to automated tools? 49 days, which Roldan called the main downside of manual testing compared to automated tools that run in minutes or seconds. ▶ Watch (39:00)
How did Fluid Attacks arrive at using CVSS as an exponent for CVSSF? A data scientist on the team proposed it after the company repeatedly faced questions about how to compare risk exposure between applications without the linear-score distortions of plain CVSS. ▶ Watch (39:28)
Notable Quotes
basically worse than random guessing Andres Roldan · ▶ Watch (32:39)
84% which was really really really good Andres Roldan · ▶ Watch (32:24)
less than 10% of the risk exposure Andres Roldan · ▶ Watch (25:18)
Key Takeaways
- Manual pentesting found 74.8% of vulnerabilities that no automated scanner detected.
- The best automated scanner scored 36.9% on F1, below the random-guessing threshold of 50%.
- CVSSF uses the CVSS score as an exponent, making risk values additive and exponentially separated by severity.
About the Speaker(s)
Andres Roldan is VP of Hacking at Fluid Attacks, where he leads the company’s research team. He has identified and ethically disclosed 110 CVEs in open-source software, holds over 29 certifications in offensive security, and serves on the GIAC Advisory Board. He brings more than 20 years of cybersecurity experience to the role.