Why Outcome-Only Benchmarks Fail Security Agents

▶ Watch (00:58)

Benchmark scores for security agents are too thin to act on.

“Agent evaluation today is rooted in very narrow outcome only scoring which essentially hides explainability and reliability.” — Mudita Khurana

An 80% pass rate tells you nothing about why the agent succeeded those 80% of the time, what caused the other 20% to fail, or how to close that gap. Worse, it gives you no signal on whether those results will hold when you move the agent to your own infrastructure. Outcome-only grading is a dead end.

What a Passing Score Can Conceal: The SQL Injection Case

▶ Watch (01:45)

The benchmark says pass. The agent found SQL injection. But look at the SQL injection agent trace walkthrough: it matched a pattern and stopped. No taint flow traced into the method. No check on whether the input was sanitized downstream. No evidence of how it found the bug.

“security is not an isolated task right it is almost always a workflow” — Mudita Khurana

Security runs in a loop: find, exploit, patch, verify. An agent that skips taint analysis and exploitability passes stage one but carries no context forward. The patch it hands off may be incomplete or wrong. One stage succeeds; the full workflow breaks.

CLASP: Six Capability Rubrics for Agentic Evaluation

▶ Watch (04:01)

CLASP gives agents six rubric scores — reasoning, memory, planning, tool use, and two others — each rated on a brittle-to-adaptive scale. A low score means the agent executes the capability the same way every time, ignoring environmental feedback. A high score means it adjusts. The framework shifts the question from “did we get the result?” to “how did we get it?”

Applying CLASP to existing agents produced a clear finding: not all six capabilities matter equally for every task. For enumeration work, Recon agent breadth-vs-depth finding — agents that planned tool coverage wide and ran those tools extensively beat agents that reasoned deeply about each finding. Knowing which capabilities drive success for a given task type tells you exactly where to improve an agent, rather than guessing.

Using CLASP in Practice: Observe, Evaluate, Improve

▶ Watch (06:03)

Start with observability. Bake it into your workflows so you have access to the reasoning trace, tool calls, and retrieved context. Then evaluate: the lighter path is an LLM-as-judge pipeline fed the CLASP rubric and your agent’s trace. A score of 2 on planning tells you exactly where to improve. The heavier path builds test scenarios at varying complexity for each capability, drawn from evidence-centered benchmark design. Either way, you get a number, a gap, and a fix.

“don’t evaluate on isolated narrow outcomes. Dig deeper into the how of these agents” — Mudita Khurana

Q&A

Do you have prompts or structured data for the LLM-as-judge pipeline that could be open-sourced? Khurana said the paper’s blueprint covers a full benchmark (no LLM-as-judge needed) and that she has judge prompts she is considering open-sourcing. ▶ 8:46

Notable Quotes

agent evaluation today is rooted in very narrow outcome only uh scoring which essentially hides explanability and reliability Mudita Khurana · ▶ 0:58

security is not an isolated task right it is almost always a workflow Mudita Khurana · ▶ 2:40

don’t evaluate on isolated narrow outcomes. Dig deeper into the how of these agents Mudita Khurana · ▶ 8:01

Key Takeaways

  • Score agent capabilities — reasoning, memory, planning, tool use — not just final outcomes.
  • An agent that passes a benchmark via pattern matching will break multi-stage security workflows.
  • Use LLM-as-judge with CLASP rubrics to get a clear improvement path, not just a pass/fail number.