Why File Similarity Matters

▶ Watch (2:57)

MD5 and SHA256 only match identical files. Similarity detection finds variations. Cobalt Strike and LLMs make generating modified malware trivial. At Stairwell, searching over a billion unique files on tens of thousands of endpoints becomes a needle-in-haystack problem. Family attribution for threat hunting also requires similarity.

Feature-Based Similarity: Hamming, Edit Distance, Jaccard

▶ Watch (7:38)

Hamming distance counts differing bit positions. Edit distance (Levenshtein) handles offsets better but scales linearly per file. Jaccard similarity compares sets of features (n-grams, imports) and runs faster. MinHash approximates Jaccard for large corpora. Limitations: sensitive to reordering and lacks locality awareness.

Fuzzy Hashing: SSD, SD Hash, and TLSH

▶ Watch (12:48)

SSD uses a rolling hash to segment files, concatenates chunk hashes, and compares with Levenshtein distance. SD hash replaces ordered concatenation with set-based bloom filters and Shannon entropy, reducing reordering sensitivity. TLSH generates a fixed-size histogram of 256 n-gram buckets, producing a metric that supports vector search and clustering. TLSH is more robust to changes; SD hash is better at containment detection.

Learning-Based Similarity: XGBoost and Deep Neural Networks

▶ Watch (19:28)

XGBoost builds sequential decision trees for fast classification. Embeddings derived from leaf node paths enable Hamming distance comparison. Deep neural networks learn hierarchical features; removing the classification layer yields vector embeddings. Euclidean distance between embeddings measures similarity. XGBoost excels at classification; deep networks at similarity.

Experimental Results on the Ember Dataset

▶ Watch (24:42)

The Ember dataset contains static features from 800,000 labeled PE files (400k malware, 400k benign). Crowdstrike added AVclass family labels. XGBoost achieved 90% accuracy in 20-class family classification (random guess is 5%). Deep learning scored 94–95% label homogeneity for similarity (100 neighbors), compared to 75% for XGBoost and 20% for SSD. Conclusion: use XGBoost for classification, deep learning for similarity, and keep fuzzy hashing for explainability.

Q&A

How do the models handle VM-based obfuscation or encrypted VM instructions? The experiments were limited to the Ember dataset; no raw files were available, so the speaker couldn’t test obfuscated samples directly. ▶ 34:20

Have you considered using ASTOVE embeddings for binary files? The speaker didn’t have raw files to generate such embeddings but noted it could be explored with corpora like MalwareBazaar. ▶ 36:03

Which feature was most important for XGBoost? String-based features like exports and imports flagged the most in XGBoost. ▶ 37:51

How do you address false positive risk? You can tune the precision-recall trade-off in the model to favor fewer false positives at the cost of potentially missing malware. ▶ 38:14

Does dataset size correlate with accuracy? More data generally helps up to a point, but temporality matters—a model trained on 2014–2018 malware would not perform well on current threats. ▶ 38:47

Notable Quotes

for similarity deep neural networks does better than XG boost and both the machine learning models do way better than uh the fuzzy hashing model Udbhav Prasad · ▶ 32:09

we’re very thankful that u the Ember data set exists Udbhav Prasad · ▶ 25:08

hybrid systems are essential Udbhav Prasad · ▶ 33:10

no malware were harmed to make this presentation, I think Udbhav Prasad · ▶ 39:54

Key Takeaways

  • Use XGBoost for classification and deep neural networks for similarity.
  • Fuzzy hashing (TLSH) remains valuable for explainability and locality.
  • Machine learning models far outperform fuzzy hashing on label homogeneity.