AI
Security Debt Grew While DevSecOps Failed to Ship
DevSecOps promised secure products released at the speed of DevOps. It failed. Developers don’t fix vulnerabilities. Google research found its developers spend an average of two hours per issue. The industry standard runs five to six hours. When a scanner interrupts a developer mid-feature, the finding gets kicked to the backlog. Across an organization, those cans pile into hundreds of thousands of unresolved entries. PCI 4, mandated at end of March 2025, requires documented justification for any medium or low left unfixed.
52% of Raw GenAI Fixes Were Wrong or Broke the Application
Mobb tested GPT-3 on Juice Shop and WebGoat, feeding findings from two commercial SAST tools. The results: 29% of fixes solved the problem, even if not following best practices. Another 19% left the application functional but may have introduced new vulnerabilities. One command injection fix introduced a NoSQL injection. The remaining 52% were bad. The model changed code in the wrong location or hallucinated fixes entirely.
Why Human Review Remains Mandatory for Every AI Fix
Worcel ran a live on-stage demo with a voice-enabled ChatGPT assistant. When asked whether GenAI could remediate vulnerabilities at scale, the model answered: a human must evaluate each fix, and at thousands of vulnerabilities, that doesn’t scale. Mobb’s own testing added more problems. Context windows are small or expensive. Models hallucinate JSON output when asked for structured responses. They strip code style. Counterintuitively, sending a larger file increases hallucination rate: the model aims to please and tries to fix things that aren’t there.
Deterministic AST Fixes Work But Take Months Per Pattern
The alternative is pattern-based fixing using abstract syntax trees. Parse the code to an AST, match the vulnerability report to the right location, then apply a precise transformation. The fix is deterministic: the same pattern always produces the same correct change. Mobb used Semgrep for AST analysis. The problem is build time. Each new vulnerability type or language requires manual research from security experts. The approach scales once built but grows slowly. Adding a new issue type means starting the research cycle again.
The Hybrid: AI for One Sub-Step, Algorithms for the Rest
Mobb’s working solution combines both. Take a null reference exception: the fix is an if-statement check, but object access can nest three levels deep. Writing an AST pattern to cover every variation takes a month. Instead, give the LLM the exact line and a precise instruction. Worcel’s prompt: “surround the line code with an if sentence to avoid the null reference exception.” The AI returns the if-statement. Mobb strips irrelevant additions and slots the result into the AST pipeline. The same prompt reliably produces the same output, so one review approves the entire backlog pattern.
What Two Years of AI-Assisted Remediation Actually Taught
GenAI cuts development time and should be in your stack. Companies that ban it risk falling two years behind within a market cycle. But it requires supervision. Every AI-generated fix must be validated before shipping. Attackers treat unvalidated output as an invitation. Scope the context deliberately: sending a full file produces more hallucinations than a single function. Keep AI in a narrow lane. Use it to generate one sub-step of a fix, then let deterministic code integrate the result. That constraint is what makes the output predictable enough to apply at scale.
Q&A
How does the LLM step relate to AST pattern matching? Are they the same thing? Pattern matching finds the vulnerability location; LLM generates the fix for complex patterns where no AST rule exists; AST then applies the change. ▶ Watch (35:23)
Can AI-generated fix outputs train new deterministic AST rules, removing AI from the loop entirely? No, because pattern variations are too numerous to formalize without months of engineering work per case. ▶ Watch (37:01)
What if the LLM had access to the full repository instead of just a code block? Mobb tested this and found full-file context made hallucinations worse, so they now deliberately minimize input size. ▶ Watch (39:25)
Do different programming languages produce meaningfully different results? Yes, and Mobb uses different models per language, including Mixtral and Claude Sonnet for Python, based on measured performance. ▶ Watch (40:28)
Notable Quotes
52 were just bad the code change was in a different place in the code AI completely hallucinate Eitan Worcel · ▶ Watch (12:49)
the biggest challenge is that we cannot assure the fixes will be correct a human needs to evaluate each fix to ensure it’s appropriate and if we’re looking at thousands of vulnerabilities doing that manually simply doesn’t scale ChatGPT demo (Elliot) · ▶ Watch (16:09)
you just broke your application because you replace it in one place what about all the other components that are using it Eitan Worcel · ▶ Watch (19:31)
think of the model is an overeager Junior employee that blurts out an answer before checking the facts Eitan Worcel · ▶ Watch (33:27)
Key Takeaways
- Developers spend 5 to 6 hours on each vulnerability fix, making backlog clearance at scale impossible without automation.
- Raw GenAI fixes fail 52% of the time and hallucinate more as file size increases.
- Scoping the LLM to one precise sub-step, then integrating via AST, produces consistent and reviewable output.
- Every AI-generated fix needs validation before shipping. Attackers will find what the team misses.
- Different LLMs perform better on different languages. Matching model to language improves output quality.
About the Speaker
Eitan Worcel is the co-founder and CEO of Mobb, a Black Hat StartUp Spotlight winner. He has over 15 years of experience in application security, working across developer, product management, and business leadership roles.