AI
Why Query Generation Is the Right Automation Target
Datadog’s threat hunters search massive log volumes across many sources with varying schemas. Figuring out which queries surface anomalous data is tedious and slow. It is also high-stakes: those queries eventually become detection rules. Log data is text, which makes query generation an obvious fit for LLMs. That decision — automate query generation first — shaped everything that followed.
V1: What a Naive Single-Agent Approach Gets Wrong
Datadog started with a single GPT-4.1 agent, web search, and the AWS Knowledge MCP for CloudTrail schema docs. Responses averaged 5 seconds. The syntax error rate hit 25%. Fields were wrong, values were irrelevant, and the agent ignored the tools it was given.
“We concluded that LLMs are bad at threat hunting and we had to go back to the basics and understand what their capabilities actually were.” — Arthi Nagarajan
The failure had three layers: low syntax accuracy, low semantic accuracy (valid queries that answered the wrong question), and low relevance from ineffective tool use. LLMs are good at synthesis, pattern matching, and translation. Threat hunting, as designed, maps to none of those cleanly.
The Schema Problem: Why Log Data Breaks LLM Context
Datadog’s log environment has 450-day lookback windows and two field types that don’t behave the same way: faceted fields are indexed and easy to filter; unfaceted fields are searchable but not indexed. Capturing full schema for any source in a small token budget is structurally hard. Knowing what values exist in each field requires actually querying the data, not reading docs. Fine-tuning was the obvious next question, but Datadog had no well-curated corpus of past hunts with consistent, labeled inputs and outputs, so the effort cost was too high for the expected return. The team stayed with prompt engineering and started treating schema discovery as a problem to solve explicitly, not a gap for the model to paper over.
V2: Multi-Agent Orchestrator Architecture and Its Trade-offs
V2 replaced the single GPT-4.1 agent with a GPT-o1 orchestrator driving expert sub-agents for AWS CloudTrail and GCP Audit Logs, plus the Datadog MCP server to execute and validate queries against real logs. The MCP returned concise error corrections, which cut syntax errors by 17%. The orchestrator kept the overall hunt context while each sub-agent started fresh on every call, capping chain depth at three layers. The cost: response time jumped from 5 seconds to 6 minutes, with some edge cases hitting 30 minutes to an hour.
“Threat hunting workflows overload context. If you manage the context well, you’ll see results improve.” — Arthi Nagarajan
Semantic accuracy still suffered. Query executions timed out, results were truncated, and sampling log data was too computationally heavy for the agent to handle alone. Every handoff added latency. Delegation managed context, but it didn’t solve hallucination.
Four Approaches to Semantic Accuracy and How to Evaluate Them
Syntax accuracy came easy. Semantic accuracy did not. Nagarajan tested four approaches: external documentation (an educated guess that missed actual log schema), clustering (agents misjudged which facets to aggregate on), ingestion-pipeline pre-processing (grok parsers consumed context and drifted over time), and live sampling (accurate but computationally exhaustive, so the work shifted to tools instead). No single method solved it. The team also dropped static benchmarks after benchmarks showed 95% accuracy while hunters disagreed. Their replacement: A/B testing during real hunt weeks, with usage rate as the primary signal.
“A single number to capture the performance can be quite misleading.” — Arthi Nagarajan
Live Hunt: AWS Privilege Escalation Case Study
During an AWS privilege escalation hunt week, a hunter focused on the IAM CreateLoginProfile event. The IAM CreateLoginProfile escalation hunt walkthrough (15:21) shows the co-pilot broadening its scope from 1 day to 30 days, identifying six targets, flagging suspicious password-reset behavior, and returning a “likely benign” verdict with named users to verify — all driven by a hunter working from a concrete playbook.
The A/B test results: hunting co-pilot vs. manual+ChatGPT (17:28) put numbers on it: query iteration time dropped from 10 minutes to 5, and the tool saved 25 of 60 hunt minutes per hypothesis. The week produced two new cloud detection rules and zero incidents. On high-volume hypotheses, the tool still struggled — truncation cut into signal, and semantic accuracy fell.
Q&A
Did you use different LLM providers or local models for the LLM judge, and which agreed most with human judgement? No cross-provider comparison was done; the only comparison was reasoning vs. non-reasoning models, and they chose non-reasoning for speed after seeing no meaningful accuracy gain from reasoning. ▶ 20:24
Did you consider adding a confidence score to the agent’s output? They skipped it because the agent was always overconfident — prompted to produce accurate Lucene queries, it rated its own output as correct, making confidence scores circular and useless. ▶ 21:31
Notable Quotes
We concluded that LLMs are bad at threat hunting and we had to go back to the basics and understand what their capabilities actually were. Arthi Nagarajan · ▶ 3:26
Threat hunting workflows overload context. If you manage the context well, you’ll see results improve. Arthi Nagarajan · ▶ 9:20
A single number to capture the performance can be quite misleading. Arthi Nagarajan · ▶ 13:19
For us, usage corresponded to success. Arthi Nagarajan · ▶ 14:19
Key Takeaways
- Automate schema discovery and query iteration, not threat hunting judgment — hunters must drive the hunt.
- Cap agent chain depth at three layers; longer chains get stuck and latency compounds multiplicatively.
- Replace static benchmarks with A/B testing during real hunts — usage rate is the most honest accuracy signal.