AI
Why Long-Running Agents Break Down
Three things stop agents from running for hours. Context windows are finite, so a new session starts with amnesia. Coherence degrades mid-session (context rot), and models that near the end of their window rush to finish, what the team named “context anxiety.”
Planning is the second problem: models attempt everything in one shot or abandon half-finished work. The third is self-evaluation bias. A half-built button gets marked done. A UI with no backend behind it looks complete. Both speakers call this the most underestimated failure mode, and it shapes every harness decision that follows.
One Year of Model and Harness Co-Evolution
From Sonnet 3.7 to Opus 4.6 over one year, the threshold at which agents complete 50% of tasks with a minimal scaffold jumped from 1 hour to 12 hours. Every model release came with harness changes: checkpoints with Sonnet 4.5, server-side compaction and a 1-million-token context window with Opus 4.6. By Sonnet 4.5, the team ran 30-hour sessions, then dropped context resets entirely when Opus 4.6 held continuous sessions without anxiety.
The harness never disappears as models improve. Anthropic finds a gap, fills it with scaffolding, trains the model on that behavior, then removes the scaffolding. The cycle repeats.
Separating Generator from Evaluator
Asking an agent to review its own pull request is the canonical failure. Prabaker’s fix borrows from GANs: a generator builds, a separate evaluator grades. The evaluator uses Playwright to open live pages and write a critique. It never sees the builder’s context. Tuning a standalone critic to be harsh is tractable; tuning a builder to be self-critical is not.
Before any code gets written, the two agents negotiate what “done” means via markdown files on disk. The evaluator grades against that contract, not the original spec. Vague criteria produce vague critiques. The team settled on 27 for one app.
What the Harness Actually Changed
The prompt was “build a retro game maker.” The solo run produced a canvas, palette, and frame timeline. Arrow keys did nothing. The physics loop was absent.
The retro game maker harness vs solo run (28:58) ran six hours and cost $200. The harness output named itself RetroForge, built a 54-color sprite editor, a working physics loop, and collision detection. The evaluator caught FastAPI route-ordering bugs that passed unit tests, and a delete-key boolean bug only visible by actually playing the game. Both outputs came from the same prompt and the same model.
Adapting the Harness as the Frontier Moves
Opus 4.5 needed context resets and sprint decomposition to stay coherent. Opus 4.6 held a two-hour continuous build without them. The harness was not wrong for 4.5. The frontier moved, and the team simplified. The evaluator now runs at the end of a full generation rather than after each sprint, cutting cost roughly in half.
Five takeaways: self-evaluation is a trap, use an adversarial critic. Compaction does not equal coherence. Structured handoffs and clean context work. Subjective quality is gradable if you write the rubric down. Read the traces. That is the primary debugging loop, not running more experiments.
Q&A
Is the evaluator tuning reusable across projects or specific to each app? The goal was reusability: identify common model weak points (front-end design, for example), write calibration examples showing good and bad output, and those generalizations hold across apps. ▶ 40:19
With a 1M context window and compaction, is the RLAIF-style loop that keeps the model in a ‘smart zone’ still necessary? Opus 4.6 with compaction handles long continuous sessions well enough that the team dropped context-window rotation, though Wilson noted it depends on your use case and evals. ▶ 41:32
Can the generator-evaluator loop keep improving after initial delivery, and will the evaluator eventually give up? Prabaker said 4.6-class models were surprisingly willing to throw out ten passes of work and restart from scratch when they couldn’t hill-climb against the rubric, making a human-resume mechanism less necessary than expected. ▶ 45:45
What artifacts should persist for long-lived products where changes happen days or weeks later? Write learnings and state to a JSON file throughout the run (models are less likely to overwrite JSON than markdown), and keep a live-updating file-structure doc; those two files are enough for a human or a new Claude Code session to continue. ▶ 53:52
Notable Quotes
it’s better to fail predictably Andrew Wilson · ▶ 9:06
most people say you can’t grade taste Ash Prabaker · ▶ 21:31
self-evaluation very much a trap. Ash Prabaker · ▶ 39:01
Key Takeaways
- Split builder and critic into separate context windows; self-review produces sycophantic rubber-stamping.
- Granular contracts between generator and evaluator — 27 criteria for one app — make critiques actionable.
- Harness complexity should shrink as models improve; read traces to know which scaffolding to drop.