The Complexity That Defeats Manual Debugging

▶ Watch (2:31)

Incident.io’s AI SRE investigation fires hundreds of telemetry queries per incident. Behind it are hundreds or thousands of prompts across dozens of agents. A single report takes an engineer an hour to evaluate. The full agent graph—10 agents, 50+ tools—didn’t fit on a slide. You cannot scale manual review to thousands of daily runs across hundreds of customer accounts.

CLI Eval Tooling for Agent-Friendly Test Suites

▶ Watch (6:59)

Large YAML eval files hit context limits when coding agents tried to read them. Jones’s team built a CLI that lets agents list, add, replace, and delete test cases. Agents now follow a runbook: create a failing eval, modify the prompt until it passes, then check no other eval broke. A final pass consolidates the prompt to avoid bloat.

File Systems Beat APIs for Agent Debugging

▶ Watch (11:03)

Instead of building MCP servers or custom UI integrations, Jones exports the full interaction as a text file system. Traces, tool calls, and prompt outputs become ASCII files. Claude Code greps the file system, cross-references the code base, and pinpoints which prompt caused the failure. The agent then uses the eval runbook to fix it.

Parallel Analysis Pipelines for Thousands of Runs

▶ Watch (13:38)

The scrapbook repo starts 25 Claude Code agents in parallel, each analyzing one investigation from a batch download. They store incremental analysis in files. A second stage clusters failures by type and correlates them with code. The agent produces a report with recommended changes, then generates a PR after passing the eval suite.

Notable Quotes

evals for me are AI unit tests Lawrence Jones · ▶ Watch (4:12)

we just don’t have enough time to go through this stuff Lawrence Jones · ▶ Watch (10:37)

file systems are exceptionally good agent context Lawrence Jones · ▶ Watch (16:20)

it will save you literally days or maybe weeks of your life Lawrence Jones · ▶ Watch (16:43)

Key Takeaways

  • Export AI system traces as file systems so coding agents can grep and debug.
  • Build CLI eval tools that let agents add, edit, and verify test cases.
  • Use parallel agent pipelines to analyze thousands of runs and auto-generate fixes.