The Benign Long-Tail Problem in Training Data

▶ Watch (02:21)

Sophos’s command-line classifier was stagnating. Adding malicious labels from analysts helped, but the fix didn’t scale. The deeper problem was benign data: approved-machine sampling captures only the most common commands, like cd, leaving rare but legitimate commands untouched. ML models generalize only within their training distribution. A sysadmin command that never appears in training becomes a false positive in production.

The benign long tail needed coverage. Getting there through expert labeling was too slow. A different sourcing method was required.

How a Failed Experiment Became the Right Idea

▶ Watch (03:44)

The original plan: run anomaly detection, send flagged commands to an LLM, call the malicious ones confirmed threats. An expert analyst checked the results.

“the LLM scored an unusable 36% precision on the commands it claimed were malicious” — Ben Gelman

Anomaly detection finds weird things. Weird benign commands fooled the LLM into calling them threats. But flipping the question worked. When the LLM labeled an anomalous command as benign, it was almost always right. The failed pipeline had accidentally demonstrated a high-confidence benign labeling system.

Pipeline Architecture: Two Scales, One LLM Labeler

▶ Watch (11:04)

The pipeline runs at two scales. Full scale processes 50 million commands per day using expert hand-engineered features and isolation forest, runnable on CPU. Reduced scale samples 4 million commands, embeds them with Jina embeddings v2 base code on cheap GPU instances, and adds K-means and PCA anomaly detection for diversity. Both converge at the same step: OpenAI o3 Mini labels each anomalous command as benign or malicious. The confirmed benign anomalies feed back into XGBoost classifier training. Every component is swappable: the data, features, anomaly algorithm, LLM, and target classifier can all be replaced without changing the structure.

Anomaly Algorithms and Near-Duplicate Filtering

▶ Watch (19:21)

Three algorithms find anomalies in fundamentally different ways. Isolation forest counts random splits needed to isolate a point: outliers split off fast. Modified K-means measures Euclidean distance to the nearest cluster center. PCA computes reconstruction error when a point is forced into reduced dimensions. Diversity across algorithms catches more of the benign long tail. Before any command reaches o3 Mini, near-duplicates are filtered using cosine similarity on embeddings: commands within 0.8 similarity are dropped. In the o3 Mini labeling demo (27:30), a complex WildFly Java startup command gets a step-by-step benign verdict in seconds.

Results: AUC Gains on Hard Production Cases

▶ Watch (28:52)

Every anomaly configuration beat both baselines on both test sets. The hard test set matters most: it contains commands that broke previous production models. On the aggregated baseline, the full-scale method jumped from 61 to 89 AUC on that set. On the regex baseline, the same method went from 7 to 76. Performance saturated within a few days of data, not weeks. Adding diverse benign anomalies also left the original distribution untouched: all configurations met or exceeded baseline AUC on the easy time-split test set across the full month.

Notable Quotes

the LLM scored an unusable 36% precision on the commands it claimed were malicious Ben Gelman · ▶ 3:53

LLMs are the key point that underpins our whole process Sean Bruin · ▶ 26:21

anomaly detection excels at locating benign data in the long tail Ben Gelman · ▶ 35:34

Key Takeaways

  • Anomaly detection reliably surfaces rare, legitimate commands that approved-machine sampling never reaches.
  • o3 Mini labels benign anomalies accurately without human review, making the pipeline fully automated.
  • Adding benign long-tail data lifted AUC from 61 to 89 on the hardest production test cases.