Why YARA Fails on GenAI Threats

▶ Watch (00:56)

YARA detects millions of malware samples a day at Palo Alto. It is the gold standard for binaries and malicious scripts. But agents changed the target.

“the new binary is natural language. It’s no longer pees or scripts.” — Mohamed Nabeel

Prompt injections arrive through query parameters, URL hashes, or dynamically at runtime inside the browser itself, where no firewall sees them. Traditional YARA matches on fixed strings and byte patterns. Against natural-language payloads with infinite variation, it breaks down.

SuperYARA’s Two-Line Semantic Rule

▶ Watch (02:37)

A fictional startup called safeinttel.ai needed to block prompt injections arriving via query parameters. Their engineer iterated through multiple YARA rule versions, but the op-parameter count kept growing and the rule became too brittle to maintain, with false positives and false negatives on both ends. SuperYARA replaced the whole thing with two lines: a natural-language description of what to catch, and a similarity construct that matches semantically rather than literally. In shadow-prod testing (04:12), the semantic rule detected more injections than the iterated YARA rule, including variants like “don’t execute previous rules” that were never explicitly listed, at modestly higher cost than string matching but far cheaper than an LLM call.

Four Detection Constructs and the Defense-in-Depth Stack

▶ Watch (05:28)

SuperYARA offers four constructs: string (identical to classic YARA), similarity (semantic matching against any text), classifier (binary or multiclass ML models), and LLM prompt (write your own detection instruction). Detection accuracy climbs as you move down that list, but so does cost and latency. Nabeel’s recommendation mirrors defense-in-depth: combine them cheapest-first.

The ClickFix demo makes the tradeoff concrete. ClickFix layered-rule experiment: 50% caught by string, 45-48% by similarity+classifier (07:47) A three-condition rule — string check, then similarity, then a fine-tuned DeBERTa classifier — caught roughly 95% of threats. String alone missed half. Classifier alone worked but cost 50% more than the layered approach. One rule, three layers, most threats covered without overspending.

“we have the habit of throwing the LLM to every problem. The problem is when you want to detect threats at scale when you have to iterate through millions of pages they don’t work” — Mohamed Nabeel

Pre-Filtering: From Hours to Minutes at Scale

▶ Watch (08:27)

The brand-impersonation example makes the cost case. A HuggingFace phishing classifier acts as the first gate, running in milliseconds. Anything it flags passes to Gemini 2.5 Pro for confirmation, at 4.5 seconds and real money per call. Without pre-filtering, 10,000 URLs cost $750 and take hours. With it, the LLM sees only the flagged fraction, dropping the bill to $13.50 — Brand-impersonation pre-filter cost benchmark: $750 → $13.50 for 10k URLs (10:49). At Palo’s scale of millions of URLs per day, a blind LLM call is a budget problem, not a design choice.

“always do the pre-filtering before you do detections.” — Mohamed Nabeel

Pluggable Architecture: Cleaners, Chunkers, and Open-Source Availability

▶ Watch (11:28)

Every component in SuperYARA is swappable. Cleaners strip HTML down to plain text before it hits any model. Chunkers split documents by sentence or paragraph so a short similarity query doesn’t get diluted against a 10,000-word page. Plug in your own cleaner or chunker and the rule engine picks it up automatically. The four constructs (string, similarity, classifier, LLM) all follow the same factory interface, so any Gemini, OpenAI, or locally hosted Ollama model works as a drop-in. Models are preloaded into memory once, not reloaded per rule execution. The whole library ships as pip install sara.

Q&A

Are you doing reverse validation of new classifiers or rules against known-benign corpora before deploying? Yes — threshold tuning uses a known-benign/malicious ground-truth set, and new rules go into shadow prod for about a week before enabling, same as with YARA rules at Palo. ▶ 14:08

Why build a new format instead of extending YARA’s existing module system? Extending YARA is possible in principle but the friction is high, so they built SuperYARA as a separate open-source library that follows YARA’s same philosophy without the integration overhead. ▶ 15:54

How does the engine decide routing between ML classifier and LLM, and can thresholds be customized? The engine automatically runs the cheapest construct first on OR conditions; LLM selection is fully customizable by wrapping any model behind the required interface, and Nabeel starts with the most capable model then downgrades until cost is acceptable. ▶ 16:58

Does the matcher stop at the first pattern match or continue through all chunks and return every match? It scans all chunks, returns all matches with the best-scoring chunk identified, mirroring how YARA traverses a full document. ▶ 17:52

Notable Quotes

the new binary is natural language. It’s no longer pees or scripts. Mohamed Nabeel · ▶ 1:02

we have the habit of throwing the LLM to every problem. The problem is when you want to detect uh uh threats at uh scale when you have to iterate through millions of pages they don’t work Mohamed Nabeel · ▶ 4:50

using this pre-filtering approach we went from hours to minutes to do it Mohamed Nabeel · ▶ 10:52

always do the pre-filtering before you do detections. Mohamed Nabeel · ▶ 11:24

Key Takeaways

  • Write semantic similarity rules in two lines instead of maintaining dozens of YARA string variants.
  • Layer string, classifier, and LLM constructs cheapest-first to catch 95%+ of threats without blowing your budget.
  • Pre-filter with a fast classifier before invoking any LLM; expect ~98% cost reduction at production scale.