AI
Why autonomous cyber defense demands a new kind of trust
Attackers are moving from manual labor to managing AI agents that run attacks at scale. Organizations with two IT security workers — a hospital, a small company — cannot possibly cover the full threat surface. They will need autonomous defenders. So will large enterprises: anyone who has worked in security has felt understaffed. The pressure lands in the same place for all of them.
That creates a new kind of demand. A defender agent will need to quarantine the CTO’s account, patch a production codebase, or shut down a compromised host, autonomously, without a human in the loop for every call. An insider threat alert fires at 3 a.m. and the agent says,
“Just trust me, bro. Uh, I investigated the alert. You don’t have to worry about it.” — Joshua Saxe
Trusting that answer is the problem Saxe’s talk is about. The hard part is not building the agent. It is earning the confidence to let it act.
Why classical ML metrics break down in cybersecurity
Precision, recall, F-score, ROC curves — all of these assume you have an oracle that produces correct labels. Security doesn’t have that. SOC analysts disagree on whether an alert was worth investigating at a double-digit rate. Access reviewers disagree at a double-digit rate on who should reach a sensitive database table. Saxe ran a simulation to show what that noise costs: add a 1–3% label-flip rate to your evaluation data and a perfect model starts looking broken. Measured accuracy collapses. Above a certain noise level, you can’t tell whether the system improved at all. The labels are always that noisy in security. That’s the environment, not a fixable data problem.
Reframing evaluation: judge reasoning, not just outcomes
The fix is to stop evaluating agents the way you’d grade a multiple-choice test. Saxe’s reframe: assess them as you would a human hire. When a new security engineer investigates an access request, you don’t just ask whether they approved or denied it. You ask whether they gathered the right evidence, reasoned from first principles, and justified their decision under ambiguity. An AI agent that generates 100,000 tokens of reasoning and tool calls before producing a well-explained SOC decision deserves the same treatment — not reduction to a single bit of error information compared against a noisy label.
“many times we find that agents are actually right against human labelers.” — Joshua Saxe
Building a rubric-based eval program and shipping with confidence
Define the rubric first: evidence gathering, policy understanding, first-principles reasoning, auditable logs, and decision accuracy. Grade agent trajectories against it. Once you can score those dimensions at scale, train an LLM judge to automate the grading. Set a deployment bar leadership has agreed to, hill-climb until the agent clears it, then ship and monitor for drift. Saxe is blunt about the cost:
“it takes at least 50% of a team’s time to do evaluation, but you actually move like 10 times faster.”
The alternative — no evals, just vibes — means engineers tweaking prompts and claiming it works on their machine. That is not a shipping strategy.
Q&A
Have you looked at improving your LLM judge with negative judgments from humans — when humans disagree more with the judge, does that work better or worse? Saxe confirmed yes in principle — a Bayesian model should discount labels with high inter-rater disagreement and weight labels where reviewers agree strongly, though he deferred the full discussion to after the talk. ▶ 19:24
Is there a material difference in evaluation methods for security versus non-security AI models, or can you apply the same methods here? The underlying statistics are standard, but security requires deep domain experts to define the rubric dimensions — at the same caliber as the people you’d hire to staff your security team. ▶ 20:43
Notable Quotes
Just trust me, bro. Uh, I investigated the alert. You don’t have to worry about it. Joshua Saxe · ▶ 2:11
many times we find that agents are actually right against human labelers. Joshua Saxe · ▶ 13:24
it’s all engineers, you know, tweaking their agents and saying it works on their machine. Joshua Saxe · ▶ 16:57
it takes takes at least 50% of a team’s time to to do evaluation, but you actually move like 10 times faster. Joshua Saxe · ▶ 17:28
Key Takeaways
- Treat autonomous security agents as decision-makers under uncertainty, not oracle-based classifiers.
- Noisy security labels make binary metrics misleading — build rubric-based, multi-dimensional evals instead.
- Invest 50% of team time in evaluation; the return is ten times faster shipping and a defensible deployment bar.