Why Classification Is Core to Security Operations

▶ Watch (00:25)

At a managed detection and response company like Expel, classification is the job. Sorting malicious from benign, flagging the possibly-suspicious middle ground, triaging what gets acted on first. Xenia Mountrouidou has spent 15 years building predictive models for security data, and one question kept coming back: can generative models do what traditional ML does, or do they just look like they can?

Experiment Results: Traditional ML Beats LLMs on Networking Data

▶ Watch (01:37)

Mountrouidou tested LLMs as zero-shot classifiers against a trained traditional ML model on labeled networking data. The traditional model won on both precision and recall — it was the most sensitive to malicious traffic and the most precise.

“womp womp, the traditional ML model wins.” — Xenia Mountrouidou

The zero-shot result is still worth noting. No training data, no fine-tuning — just a prompt. That the LLMs classified at all on raw networking features points to room for improvement, especially with fine-tuning or on text-native data like emails.

Why LLMs Struggle With Non-Text Data and Where They Could Improve

▶ Watch (02:57)

Packet captures are not text. Mountrouidou had to engineer features manually — protocol headers, payload fragments, connection timelines — before the model could attempt classification at all. That structural mismatch explains the performance gap more than model quality does. Fine-tuning could help, but Mountrouidou’s early experiments with specialized embeddings produced mixed results: sometimes a generalist model with broad embeddings beats a domain-specific one.

“traditional models can find those boundaries. And LLMs were not meant for that.” — Xenia Mountrouidou

The more promising path is ensembles. Mountrouidou has ongoing research combining traditional ML and LLM outputs, treating the two as complements rather than competitors.

Q&A: Input Engineering and Where LLMs Can Actually Win

▶ Watch (03:48)

The LLM input was Mirai botnet packet captures, not text. Mountrouidou hand-engineered features from protocol fields, payload, and timelines before prompting the model, which made it one of the hardest possible classification tasks.

“Oh, just give it to a language model. It’s going to classify it. It’s going to find evil.” It’s like, “No. For networking, it won’t.” — Xenia Mountrouidou

Email classification is a different story. A router experiment on phishing datasets, XGBoost first and then LLM, beat either model alone. Claude Opus 4.6 also performed well on small samples (around 100 emails) but degraded as the dataset grew. The pattern held: traditional models find decision boundaries; LLMs don’t.

Q&A

What was the input for your LLM classifier? Packet captures from the Mirai botnet dataset, requiring manual feature engineering (protocol, payload, timelines) before the zero-shot LLM could attempt classification. ▶ 3:57

Is there a dataset or classification type where LLMs would beat traditional ML? A router ensemble (XGBoost first, then LLM on uncertain cases) beat either model alone on phishing datasets, but a standalone LLM has not beaten traditional ML in any of her experiments. ▶ 5:29

Notable Quotes

My friends call me Dr. X. I have a very long last name. Xenia Mountrouidou · ▶ 0:09

womp womp, the traditional ML model wins. Xenia Mountrouidou · ▶ 1:47

“Oh, just give it to a language model. It’s going to classify it. It’s going to find evil.” It’s like, “No. For networking, it won’t.” Xenia Mountrouidou · ▶ 4:59

traditional models can find those boundaries. And LLMs were not meant for that. Xenia Mountrouidou · ▶ 7:22

Key Takeaways

  • Zero-shot LLMs lose to trained traditional ML on precision and recall for network traffic classification.
  • Packet captures require careful feature engineering before LLMs can attempt classification — they are not text.
  • Route first through traditional ML, then pass uncertain cases to an LLM — hybrid ensembles beat either model alone.