AI
The Fuzzing Loop: What Breaks and How Orion Fixes It
Fuzzing looks simple: seed inputs, mutate, execute, measure coverage, keep interesting inputs. In practice, every step is brittle. Bad harnesses crash or waste cycles. Missing seeds get rejected. Interpreting crashes takes manual work. Orion replaces each manual step with a dedicated agent that outputs the next artifact. The pipeline runs sequentially — interface analysis, harness generation, seed generation, fuzzing, bug analysis, patch generation. If an agent fails, it self-corrects using compiler errors or LLM judge feedback. The goal is a single push-button flow from code to patch.
Building Agents That Can Be Trusted
LLMs hallucinate. Orion compensates with a validation loop around every agent. Each output passes through deterministic tools first: the compiler for syntax, the reproducer for bug fixes, coverage measurements for seeds. A secondary LLM judge checks semantic correctness against guidelines. Only after all checks pass does the agent’s output move forward. For harness generation, the dependency analysis agent must identify setup and teardown functions from raw code without documentation. That agent alone drives a 71% pass rate on the metric that counts all required calls.
Benchmarking: 95% Accuracy and 70% Patch Acceptance
Orion’s benchmark suite uses human-annotated ground truth for three targets: NVIDIA GPIO driver, and two open-source user-space applications. Seven metrics evaluate interface analysis; the worst two — dangerous expressions and sink functions — still score above 90% combined. Harness generation scores 71% on including all required function calls, a hard task even for humans. The team compared manual vs. automated effort: interface analysis saves days to weeks, harness generation saves the most time. Patches pass validation loops at roughly 70% after initial iterations; the first version often simply deletes the crashing code.
Lessons for Anyone Building Fuzzing Agents
LLMs need the right context — too much overwhelms, too little causes hallucination. Infrastructure that parses codebases into call graphs and dependency lists is mandatory. Tools are not obsolete: compilers, sanitizers, and scripts are better at deterministic tasks than any model. The team advises keeping agent tasks simple; a script often beats a multi-step agent. For benchmarking, use LLM judges only when ground truth is unavailable, and always verify with a deterministic tool. Orion remains deeply coupled to NVIDIA’s build and fuzzing infrastructure, so open-source release will be modular rather than a full drop-in.
Q&A
Can Orion fuzz languages other than C? Yes, as long as the fuzzer supports the language. ▶ 35:30
Why GPT-4.1 over other models? Both GPT and Claude are close; GPT gave slightly better results in harness generation at the time of the demo. ▶ 36:08
Can Orion run on local hardware? Currently cloud-only, but fine-tuned smaller models are in development for local deployment. ▶ 37:32
What is the patch acceptance rate after validation? Around 70% of patches that pass the validation loop are considered good; initial versions were about 30%. ▶ 38:36
How much performance gain comes from the seed generation agent? The team always runs with seeds and has not measured the delta, but seeds are validated for coverage and crash-free execution. ▶ 40:28
Notable Quotes
we should never trust them, never trust them blindly and always run through the validation loops. Max Bazalii · ▶ 4:08
the more simple that task is, the more reliable your agent will be. Marius Fleischer · ▶ 32:20
be ready to build a lot of wrappers and glue code around existing tools. Marius Fleischer · ▶ 33:11
I think each side doing what they designed it to do. Max Bazalii · ▶ 30:42
it’s still software. So all the things that we learned about how to benchmark software, that still applies here as well. Marius Fleischer · ▶ 31:48
Key Takeaways
- Orion replaces manual fuzzing workflow steps with agentic LLM pipelines that self-correct using deterministic oracles.
- The system found 100 new bugs in NVIDIA’s drive stack over 18 months.
- 95% metric accuracy on interface analysis and 70% patch acceptance show viability for production deployment.
About the Speakers
Max Bazalii is a Principal Engineer on the NVIDIA DriveOS Offensive Security team, where he leads AI automation projects focusing on software security and formal verification. Prior to joining NVIDIA, he specialized in the security research of mobile operating systems. He has authored numerous publications and delivered technical presentations on jailbreaking Apple platforms, including the first public jailbreak of the Apple Watch. He also served as a lead security researcher on the Trident exploits during the first Pegasus iOS spyware incident. Max holds a Ph.D. in Computer Science, with a focus on software security.
Marius Fleischer is a security engineer at the NVIDIA DriveOS offensive security team. He is passionate about applying advancements in AI to tackle security challenges and has a deep interest in low-level software. Previously, Marius worked at the Security Lab of UC Santa Barbara, where he contributed to advancing the state-of-the-art in automated vulnerability detection for operating system kernels.