AI
Why Static Benchmarks Fail Adaptive Agents
Traditional AI evaluation relies on static benchmarks and offline tests. Vincent Koc argued this misses chaos engineering. Agents are not static software, but we test them that way. When an agent’s customer base shifts, the same tests no longer apply. Koc pointed to 2025’s reality: code is cheap, models are powerful, and agent harnesses like OpenClaw adapt themselves. Static benchmarks cannot keep up with that speed. The result is a calcification problem — eval suites grow brittle while agents evolve.
From Prompt Engineering to Intent Engineering
Koc traced three eras: prompt engineering (bashing words), context engineering (steering agents via RAG and tool calling), and intent engineering (machines self-optimizing based on user intent). OpenClaw’s harness and Claude’s Codex adapt to each user. That personalization makes evaluation harder — one person’s good answer differs from another’s. The shift demands evals that measure outcomes, not fixed question-answer pairs. Intent engineering turns the evaluation target into an end state rather than a dataset.
Telemetry in the Loop and Self-Correcting Agents
Koc described telemetry-in-the-loop: agent harnesses aware of cost, errors, and performance can self-correct. He cited a paper on this concept. Instead of predicting failures, evals should use live traces to drive adaptation. The agent can regenerate test suites from its own traces. This turns evaluation from a static snapshot into an always-on service that co-evolves with the agent. The harness heals itself when something breaks.
The 80/20 Rule for Malleable Evals
Koc said 80% of agent behavior is predictable and can be defined as static test suites. The remaining 20% is the dangerous part — unusual user requests, edge cases that break the business. That 20% is always shifting. Evals must become self-optimizing agents that monitor and adapt to the 20% automatically. He called this treating evals as code, not data. The goal is a living evaluation system that adjusts as the agent changes.
Notable Quotes
evals are a little bit dead. Um and it’s a little bit of a joke, but there’s a little bit of truth to it as well. Vincent Koc · ▶ Watch (1:39)
our AI applications are not static, but we’re treating them like they’re static software Vincent Koc · ▶ Watch (4:21)
our evals become like what is the end state that we want to get to and then we just let the machines do the work Vincent Koc · ▶ Watch (13:13)
80% is like the static stuff, but that 20% is always going to keep changing and it’s that 20% that’s going to mess up your business Vincent Koc · ▶ Watch (13:36)
Key Takeaways
- Static benchmarks cannot adapt to agentic AI’s evolving behavior.
- Intent engineering shifts evaluation from fixed tests to outcome-based measurement.
- Evals must become self-optimizing agents that monitor and adapt the 20% of edge cases.