AI
The Prototype-to-Production Gap
Most AI POCs stall at the same point: two or three demos work, then production breaks everything. The models are not the problem. Traditional software engineering is deterministic (1+1=2), but LLM systems are not, and the operational rigor required to manage them has not kept up. Logs tell you what happened; they do not tell you why the system behaved differently on Tuesday.
The fix is not better tooling. It is treating prompts, tool calls, and failure modes with the same discipline as any other production system: versioned, tracked, and continuously scored.
Trainline’s Production Multi-Agent System
Trainline sells 6.3 billion tickets across 27 million active users and runs a multi-agent travel assistant that does more than answer questions. It books alternative trains when a service is cancelled, processes refunds end-to-end, and hands off to human support when needed. Every one of those paths involves real money and real SLAs.
Switching models to cut OpenAI and Anthropic spend was a recurring need, but before BrainTrust, Trainline had no scores to verify that a cheaper model matched its predecessor.
“brain trust has helped us a lot uh in shipping fast.” — Mayan Soni
Building a Five-Stage Agentic Pipeline
The workshop built a support triage agent in five sequential stages: context collection, triage with tool calls, policy review, customer-facing reply generation, and escalation routing. A single-shot LLM call handles simple cases but cannot expose where reasoning breaks down.
“just because it works in the demo doesn’t” — Giran Moodley
Breaking the monolith into stages makes failures addressable. When a billing escalation misclassified as low-severity, the agent’s pipeline showed exactly which stage produced the wrong confidence score, making the fix a one-step prompt change rather than a full rewrite.
Instrumenting with End-to-End Tracing
Adding BrainTrust tracing wraps existing LLM SDK calls with parent and child spans, requiring no architectural changes. Every invocation records prompt tokens, output tokens, cost, latency, and metadata, all queryable in the UI. BrainTrust uses a custom database called BrainStore, built specifically for high-cardinality semi-structured trace data, and writes are non-blocking, so the instrumentation does not slow the pipeline.
He walked through the live tracing demo in BrainTrust (52:47), drilling into nested spans for the triage specialist stage, showing the waterfall timeline and exactly how long each tool call took.
Evaluations and Golden Datasets
A golden dataset of 10 support tickets, each with expected outputs and metadata, gives the evaluation something to measure against. Two scoring functions cover different cases: deterministic checks (schema valid, escalation reason present when required) run cheaply on every trace; LLM-as-judge scores handle nuance like brand tone and policy compliance that deterministic logic cannot capture. The rule: if you cannot write it as a boolean check, use a model.
The experiment view tracks every run, so a prompt change does not ship unless it improves the score without regressing other test cases.
Closing the Loop: Managed Prompts and Remediation
Pushing prompts, tools, and parameters into BrainTrust’s managed infrastructure lets non-engineers change a model or edit a prompt without touching code. A product manager can update a parameter, save a versioned comment, and the running application picks up the change on the next call. Online scoring then applies those same evaluation functions to live production traces, sampling at 100% early on, tapering to 5 to 10% once a baseline is established.
He demonstrated the failure replay and prompt remediation (01:20:45), showing a CFO-invoice ticket that the agent misrouted, tightening the triage prompt, and watching the experiment diff confirm the score recovered.
“observability is table stakes at this” — Giran Moodley
Q&A
How can you run online scoring when you don’t have ground truth? Flag those traces as edge cases, push them into your evaluation dataset, and replay them through the judge to build ground truth incrementally. ▶ 1:18:30
Notable Quotes
brain trust has helped us a lot uh in shipping fast. Mayan Soni · ▶ 22:14
observability is table stakes at this Giran Moodley · ▶ 1:35:14
just because it works in the demo doesn’t Giran Moodley · ▶ 34:06
Key Takeaways
- Break monolithic LLM calls into specialist stages so failures surface at the exact step, not the whole pipeline.
- Build a golden dataset of edge cases before production; deterministic checks are cheap enough to run on every trace.
- Online scoring with a sampling rate of 5 to 10 percent closes the loop between production failures and prompt fixes.