BinWhisper: LLM-Driven Reasoning for Automated Vulnerability Discovery Behind Hall-of-Fame

by Qinrun Dai

Day 2 · Briefings · Exploit Development · 30 min · 3 min read

AI Security

TL;DR

Qinrun Dai's team built BinWhisper, an LLM agent pipeline that finds memory bugs in stripped ARM64 binaries without source code. The key step is data structure reconstruction: a custom IDA plugin extracts allocation sites, then an agent traces offsets across member functions and fills in field names before any vulnerability analysis runs. Applied to Samsung's video engine service, the tool found five out-of-bounds write bugs confirmed as true positives. OpenAI o3 found all five with high confidence; any result below 80% confidence was discarded.

Manual Binary Auditing Costs That Motivated Automation

▶ Watch (02:11)

CVE-2024-34587 was a single memcpy trusting a 16-bit attacker-controlled length field. Finding it required tracing call contexts and data structures across modules to verify the buffer size. The bug was simple. The reconstruction was not.

Stripped ARM64 binaries make it harder still. Comments, type names, and symbols are gone. Reverse engineers track offsets by hand, resolve indirect calls, and write data structures in a debugger. That reconstruction often takes longer than proving the bug. That cost is what pushed the team toward LLM automation.

Why ‘Maybe’ Answers Break at Scale

▶ Watch (06:32)

Paste a function into a chatbot and ask “is it vulnerable?” and you get maybes. Auditing hundreds of functions, you need yes or no. The IDA plugin Jetto renaming demo (05:33) makes code easier to read by renaming anonymous variables, but renaming alone does not confirm a bug.

The fix: supply exact preconditions. State that the receive buffer is 1,024 bytes, the RTCP packet buffer is 1,600 bytes, and the length field is an attacker-controlled 16-bit value. With those bounds in the prompt, the model flags the overflow with high confidence instead of hedging.

The BinWhisper Pipeline on Stripped ARM64 Binaries

▶ Watch (07:49)

BinWhisper feeds stripped SO files into a decompiler with custom plugins, then attaches Frida to a real device to resolve indirect calls and verify memory layouts. Three agents follow: one locates packet receive and parse routines by walking the global call graph, one reconstructs data structures, one runs vulnerability analysis with fully populated preconditions.

The CTransportManager data structure walkthrough (13:22) shows this concretely. The agent traces constructor offsets, follows member functions, and renames raw hex fields to names like pc_samsung_buffer. By the time the vulnerability analysis agent starts, the reconstructed structures match what source code would provide.

Real Bugs Found and Model Benchmark Results

▶ Watch (18:58)

Against Samsung’s video engine service, BinWhisper found five out-of-bounds write bugs, SVE-2024-1490 through a cluster of five. All shared the same pattern: a counter increments on each packet without a bounds check, eventually overwriting adjacent buffers. The bugs are simple. That is likely why LLMs found them.

Across four model vendors, OpenAI o3 was the most stable, finding all bugs with high confidence. DeepSeek R1 matched the detection rate but took up to 21 hours on the same job. Any finding below 80% confidence was dropped. After testing, every case above that threshold was a confirmed true positive.

Notable Quotes

The accuracy is 90% or 95% Qinrun Dai · ▶ 18:12

Key Takeaways

Data structure reconstruction before LLM analysis eliminates false positives from missing buffer-size context.
Confidence scores below 80% reliably predict false positives; drop them before human review.
o3 found all five bugs with high confidence; DeepSeek R1 reached the same results but took up to 21 hours.

Filed under

AI Security