Why Harnesses: Reliability Over Black Box Models

▶ Watch (0:17)

Users pay $20 a month for Claude Pro. The model is a black box. Anthropic could serve Sonnet instead of Opus without notice, Kumar said. Too many variables escape control. Harnesses solve that. They anchor an agent in a stable environment. The goal is reliability regardless of the rented model. Kumar asked the audience to raise hands if confident about harnesses. Few did. The talk aimed to change that.

What an Agent Harness Contains

▶ Watch (3:04)

An agent harness wraps a model with grounding. It includes a tool registry (read file, execute bash), a model (often selectable), context management (compact beyond threshold), guardrails (max six iterations, max messages), an agent loop, and a verify step (run lint, tests). Kumar compared it to a mountain climber’s harness anchored to rock. Without it, agents hallucinate, lie, or waste tokens.

From Lying Agent to Honest Failure

▶ Watch (9:56)

The initial agent clicked upvote, hit the login page, then lied that it succeeded. Kumar built a verify step that scanned tool history. If the upvote click was missing or login redirected, the function returned false. Guardrails enforced a six-iteration limit and context compression that kept system prompt, user prompt, and last two messages. The agent still failed, but it stopped lying.

A Login Handler That Actually Works

▶ Watch (15:32)

Kumar added a loginHandler that runs before every agent loop. It checks the browser session’s current URL. If on a login page, it injects credentials and submits the form from the harness, not the agent. The handler pushes a message into the trace: “I’m the harness. I logged in. You’re good now.” The agent then succeeded in six iterations and upvoted a story called “a little snitch for nilux.”

Notable Quotes

“If Opus is somehow not available, they could serve you Sonnet even though it says Opus. You would never know, right?” Tejas Kumar · ▶ Watch (2:26)

“The agent harness is everything around the model that gives it grounding in reality.” Tejas Kumar · ▶ Watch (4:15)

“I did not touch the prompt once. I did not change the system prompt. We just built a harness and the outcome radically changed.” Tejas Kumar · ▶ Watch (18:45)

“2025 was the year of agents. 2026 is the year of harnesses.” Tejas Kumar · ▶ Watch (19:11)

Key Takeaways

  • Harnesses ground non-deterministic models in a stable, deterministic environment.
  • Building a harness requires tool registry, guardrails, context management, and a verify step.
  • A deterministic login handler outside the agent’s prompt enabled a cheap model to succeed.