AI
Why Standard Benchmarks Miss Vibe-Coding
Standard benchmarks like SWEbench and HumanEval check whether generated code patches pass existing tests. That protocol assumes tests exist. Vibe-coders don’t write tests. They start from a natural language product spec and expect a working app in return, with no framework choices made. The functional correctness gap Catasta identifies is specific: does the app do what the user asked? Passing test suites doesn’t answer that question. Building a benchmark that does requires rethinking the evaluation surface entirely.
VibeBench: A Zero-to-App Benchmark Built on Real Sessions
VibeBench, open-sourced and announced at this talk, uses 20 real-world Replit sessions as product requirement documents. An AI harness builds each app from an empty repo, and an AI evaluator scores the result. This lets the benchmark run on every merged PR, not just weekly. Five input pairings test increasing complexity: single-shot zero-to-one, feature-on-reference, feature-on-agent-built (slop-on-slop), and parallel task decomposition with merge. Results show a roughly 2x performance gap between frontier and open-weights models, with slop-on-slop as the hardest scenario.
An Evaluator Agent That Doesn’t Know the Implementation
The evaluator agent that powers VibeBench is framework-agnostic by design. It reads the codebase, opens a browser, and follows a natural-language test plan with steps like logging into an admin dashboard with a specific account or clicking a toggle. Any step failure gets collected and contributes to a final score. SWEbench always had fixed repositories and known task hardness. VibeBench’s surface is entirely open. Building an evaluator that handles arbitrary implementations took several months of work.
AB Tests, Sentiment Signals, and the Limits of Automation
Replit collects millions of sessions daily. AB testing is the mechanism for staying honest about what offline benchmarks miss. Metrics include run duration, cost, and user sentiment derived from prompt analysis. A recent AB test showed run duration up 7% but cost down 8%, with mixed sentiment signals. AB results rarely deliver a clean answer. Human product judgment makes the final call. Failure clusters are found by embedding traces and running an LLM classifier rather than log-based regex, and cluster configurations retrain every night.
Telescope: From Production Trace to Shipped Fix
Telescope is Replit’s internal loop that closes from problem discovery to shipped fix. It discovers failure clusters from production traces, cuts a PR using a coding agent, then reruns VibeBench as a gate. If the score drops 10 points, the change is rejected. Controversial changes go to AB test. One example: the agent started fixing environment setup timing problems before the environment was ready. Log-based monitoring missed it. Semantic trace clustering found it, and the fix shipped without needing an AB test.
Q&A
Why open-source VibeBench rather than keep it as a private internal eval? Catasta said he doesn’t believe in competing on evaluations, comes from a research background, and that shared evals help everyone build better models and products. ▶ 22:24
What advice would you give teams trying to build something like Telescope? Catasta recommended trying again if earlier attempts stalled: frontier models now handle long-context trace analysis well enough to give sophisticated feedback, and collecting every available signal, including product feedback forms alongside traces, makes a material difference. ▶ 23:38
How does an AI engineering team develop product taste? Catasta said taste builds over time through shipping and is tightly coupled to the actual user base; teams building for non-technical users will make near-opposite decisions from teams building for software developers. ▶ 26:17
Notable Quotes
building on shaking foundations Michele Catasta · ▶ 11:44
engineer all the time, but 90% of it is Michele Catasta · ▶ 18:10
better agent every single day. Michele Catasta · ▶ 21:31
Key Takeaways
- Vibe-coding evals need functional correctness checks, not just test-suite pass rates.
- VibeBench shows slop-on-slop scenarios are roughly the hardest eval configuration.
- Semantic trace clustering finds failure modes that log-based monitoring misses entirely.