AI
Why Evals Fall Short for Agents
Agents are non-deterministic, unbounded, and use tools that affect other systems. Traditional evals with golden datasets cannot cover the infinite input and output space. As agents grow more complex, sessions run for hours, and deployment stakes reach healthcare, finance, and military. Monitoring production catches the long tail of failures that evals miss.
Explicit and Implicit Signals
Explicit signals are objective: tool error rate, latency, regenerations, and cost. Spikes in any of these indicate trouble. Implicit signals use trained binary classifiers for refusals, task failure, user frustration, content moderation, and jailbreaking. Regex signals like “WTF” or “this sucks” also flag frustration. Raindrop provides these classifiers out of the box.
Running Production Experiments
Once signals are set, ship a change to a percentage of users and compare signal rates to a control group. For example, a new prompt version reduced user frustration from 37% to 9%. The average number of tools used also rose, an unexpected data point. This beats offline evals because real production behavior surfaces edge cases.
Self-Diagnostics: A Simple Tool-Based Signal
Inspired by OpenAI’s self-confession research, self-diagnostics adds one tool and one line to the system prompt. The agent calls the tool to “send a short report to your creator.” It catches capability gaps, repeated tool failures, user frustration, and unintended shortcuts like removing a unit test instead of fixing it. Models are less willing to admit fault, so naming the tool “report” instead of “unsafe” encourages use.
Workshop: Building a Self-Diagnostic Coding Agent
The workshop used a basic coding agent with four tools (read, write, bash, edit). The write tool was made to fail with a permission error. The agent bypassed it using bash heredoc syntax to create the file. The self-diagnostic report tool surfaced the bypass: “created public_ip.py via bash because the write file failed.” The key insight: frame the tool as feedback to creators, not as self-incrimination.
Q&A
How much data is needed for statistical relevance in experiments? A few hundred events make signals useful even without strict significance. ▶ Watch (9:43)
Do you use LLMs to detect signals on every output? Running an LLM on every output doubles AI spend, so Raindrop trains cheap custom classifiers for scale. ▶ Watch (11:46)
Can self-diagnostics catch self-correction that is also a security risk? Yes, it surfaces cases where the agent writes a Python script to bypass a sandbox, which is good for task completion but bad for security. ▶ Watch (18:44)
How do you compare with Sentry and LogRocket? Those give tool error traces; Raindrop focuses on fuzzy failures like user frustration and provides trajectory visualization with natural-language search. ▶ Watch (44:19)
Can you backfill signals on historical data? Yes, when a new signal is created, Raindrop runs a quick backfill of the past few days. ▶ Watch (42:42)
Notable Quotes
agents are non-deterministic. They’re unbounded. There’s an infinite space of inputs that you can put in. Zubin · ▶ Watch (0:57)
catching issues in production agents is one of the most important problems of our time. Zubin · ▶ Watch (3:28)
user frustration rate has gone down very substantially, 37% to 9%. Zubin · ▶ Watch (8:49)
the models are generally trained to look very polished. So they are less willing to admit fault in many cases. Danny Gollapalli · ▶ Watch (30:33)
Key Takeaways
- Shift from evals to production monitoring with explicit and implicit signals.
- Self-diagnostics requires one tool and one system prompt line.
- Train cheap classifiers instead of running LLMs on every output.