AI
Why the Linux Kernel Is the Right Target
The Linux kernel gets 1,400 emails per day on its mailing list. Volunteer maintainers drown. Patches get lost. There’s no centralized tracking, and CI systems don’t coordinate with each other. More CI doesn’t mean better CI. That infrastructure gap is why kernel patches still slip through in 2025.
It’s also why the kernel is a good target for ML research. The fixes tag convention — where a later commit references the earlier commit that introduced a bug — produced 125,000 labeled vulnerability examples. That dataset is what makes training possible.
How Long-Lived Bugs Survive Undetected
13% of Linux kernel bugs hide for over five years. A 19-year bug, introduced in August 2006 and fixed in August 2025, is the extreme case. Race conditions survive the longest: triggering them requires precise timing, and the right conditions rarely appear in testing. Reference counting errors don’t crash immediately — they leak memory slowly, sometimes unnoticed for months. Missing null checks get optimized away by the compiler before anyone can catch them.
Recent bugs look faster to fix only because the data is right-censored. A bug introduced in 2022 hasn’t had time to become a decade-old survivor yet.
Dumb Baselines That Beat Fancy Models
Before training any neural model, Qu tested a baseline built from Three-number baseline AUC 0.779: lines added, lines removed, and files changed. No neural net. No code parsing. That triplet scored AUC 0.779 on 125,000 labeled kernel commits — because vulnerable commits are rare enough that guessing “safe” almost always wins.
“if your model cannot beat these simple dumb baselines it’s just a fancy tokenizer” — Jenny Guanni Qu
A decision-tree variant (B8) added 118 hand-engineered diff features and still used no neural network. The implication is direct: class imbalance inflates scores, and any model that cannot clear these bars tells you nothing about actual detection.
What Actually Improved Detection
At 512 tokens, transformers read commit messages, not code. Qu fixed that by expanding the context window to 8,000 tokens, giving the model enough room to see full diffs. AUC climbed. She also found that vulnerability patterns decay — models trained on older commits predict recent bugs less accurately, so retraining frequency matters. The biggest surprise: better data beat better architecture. Loading the training set with easy safe examples first let the model learn obvious safety before tackling subtle bugs.
“the model needs to learn what is obvious is safe before learning what is subtly dangerous” — Jenny Guanni Qu
Q&A
Was there a baseline cheat you expected to fool the model but it didn’t? The opposite happened: Qu expected the three-number baseline to fail, but it scored AUC 0.779 because class imbalance (most commits are safe) lets a model guess blindly on superficial patterns and still look good on recall. ▶ 9:13
Notable Quotes
if your model uh cannot beat these like simple dumb baselines and it’s just a to a fancy tokenizer Jenny Guanni Qu · ▶ 5:30
context length unlocks code understanding Jenny Guanni Qu · ▶ 6:30
the model needs to learn what is obvious is safe before learning what is subtly dangerous Jenny Guanni Qu · ▶ 8:23
there is there’s a long like a big gap between uh crashes to actual like exploitations Jenny Guanni Qu · ▶ 12:18
Key Takeaways
- Beat simple baselines (lines added, lines removed, file count) before claiming your ML detector works.
- Expand context windows to 8,000+ tokens so models read code diffs, not just commit subject lines.
- Train on easy safe examples first; hard negatives alone confuse models that haven’t learned obvious safety patterns.