AI
LLMs as autonomous offensive security tools
“today, it is true that language models can autonomously and without fancy scaffolding find and exploit zero-day vulnerabilities in very important pieces of software” — Nicholas Carlini
Three or four months ago, that sentence would have been false. Now it isn’t. Carlini’s argument is that LLMs have crossed a threshold — not just another incremental capability bump, but something closer in scale to the internet arriving and making remote attacks possible. The 20-year balance between attackers and defenders may not survive what comes next.
Minimal scaffolding: how the bug-finding pipeline works
The setup takes three lines. Run Claude Code in a VM with permissive settings, tell it it’s playing a CTF, ask it to write the most serious vulnerability it finds to an output file. Walk away. Come back to a report.
That is the entire pipeline. Carlini’s team found that running it repeatedly against the same codebase produces the same bug each time, so they added one more line: a hint pointing at a specific file. Cycle through every file in the project and the model reviews them all. No fuzzing harness. No months of tooling. Just the base capability of the model — which is exactly what makes it threatening.
Ghost CMS: blind SQL injection found and exploited autonomously
Ghost CMS has 50,000 GitHub stars and no critical CVE in its history. The model found the first one: a blind SQL injection where user input fed directly into a query. Blind means no output visible, only timing. Carlini wasn’t sure the bug was worth reporting until he asked the model to show him the worst-case. He walked through the Ghost CMS blind SQLi live demo (07:20) in real time, running the AI-written exploit against a local Docker instance. No authentication. It read the admin API key and secret, plus the bcrypt password hash, straight from the production database. “I wrote none of this code,” Carlini said.
Linux kernel: heap buffer overflow in the NFS v4 daemon
The Linux kernel is one of the hardest targets in software security. Carlini’s team found remotely exploitable heap buffer overflows in it — a class of bug he had never found before. Carlini walked through the NFS v4 heap buffer overflow walkthrough (09:23): client A opens a lock with a 1,024-byte owner field; client B tries to claim the same lock; the server copies client A’s owner bytes into a 112-byte kernel heap buffer when sending the denial response. The overflow is only reachable through a two-client protocol interaction — fuzzing would never find it. The attack diagram on Carlini’s slides came straight from the model’s own report. The bug predates Git.
“this bug has been in the kernel since 2003, and it’s older than some of you in this room” — Nicholas Carlini
Capability trajectory: exponential improvement and what it implies
The Meter benchmark shows a clean doubling in task duration every four months — models released in the last quarter can complete work that takes humans 15 hours, succeeding roughly half the time. Carlini cited the Smart contract exploit value benchmark (14:32): Winnie and Cole’s paper found that recent models autonomously recover several million dollars from real deployed contracts, with that dollar figure rising on a log scale across model generations. Sonnet 4.5 and Opus 4.1, both less than a year old, almost never find the kernel-class bugs that models from the last three months find routinely. That gap opened in months, not years.
“you can’t just stick your head in the sand. Like these things are working really really well.” — Nicholas Carlini
The IEA predicted solar deployment for 2040; actual deployment hit that number the following year. Security planning that assumes a near-term plateau faces the same error.
Call to action: surviving the transitionary period
Carlini’s long-run view: defenders probably win. Rust eliminates memory corruption. Formal verification closes the protocol gaps. The dangerous window is now. He has hundreds of unvalidated kernel crashes sitting on his machine — bugs he cannot responsibly report yet. Soon, anyone with malicious intent will have the same pile. “The transitionary period is where I’m most worried,” he said, “and we are in the transitionary period now.” Anthropic, DeepMind, and OpenAI’s Aardvark project are all working this problem. Carlini’s ask is simple: pick one and help. The timescale is months.
“you can’t just stick your head in the sand. Like these things are working really really well.” — Nicholas Carlini
Q&A
Given that you can’t fix every zero-day, should we focus on detecting malicious intent at the model level? Carlini acknowledged the dual-use problem directly — weak safeguards block defenders but not attackers who will jailbreak anyway, and finding the right balance is still an unsolved problem that needs more work. ▶ 22:00
As this gets faster, attackers and defenders both accelerate — does the endgame mean near-zero vulnerabilities once we can find them all? Long-term defenders probably win (Rust rewrites, formal verification), but the transitionary period between now and that future is the dangerous window, and we are in it now. ▶ 24:26
Notable Quotes
today, it is true that language models can autonomously and without fancy scaffolding um find and exploit zero-day vulnerabilities in very important pieces of software Nicholas Carlini · ▶ 1:57
this bug has been in the kernel since 2003, and it’s older than some of you in this room Nicholas Carlini · ▶ 12:08
you can’t just stick your head in the sand. Like these things are working really really well. Nicholas Carlini · ▶ 16:00
if I put a safeguard in place that’s very very weak, it will only stop the good people from using the software. The bad people are just going to jailbreak the model and they’re going to still attack it anyway. Nicholas Carlini · ▶ 23:52
Key Takeaways
- Run LLMs against your own codebase now — attackers don’t need custom tooling to find critical zero-days anymore.
- Treat capability doubling time (every four months) as a planning constraint, not a talking point.
- Act on a timescale of months: the transitionary period between current risk and a formally-verified future is where the damage happens.