AI
Structured Output Extraction as an Eval Target
Colvin built a Pydantic AI agent that reads Wikipedia pages for UK MPs and extracts structured political relations. The task looks like a typical structured-output problem (find invoice lines in a PDF, find an address in an email), but models kept adding spouses and children when the question was only about ancestors. The naive one-liner prompt scored 85% against a golden dataset built by running the same extraction with Opus 4.6 and manually spot-checking the results.
Running Evals and Comparing Prompts
Running the eval against 65 test cases shows each case’s accuracy score in Logfire. The simple prompt hit 85%. A hand-written expert prompt, the kind you’d get by asking a model to improve it, reached 92%. The per-case diff makes the gap concrete: for MP Joe White, the simple prompt returned a spouse as a political relation. For Laura Kyle, both prompts wrongly flagged a four-times-great-grandfather who was Governor of Hudson Bay, a public figure but not a politician.
“generally defining your own is far better than LLM as a judge because the LLM as a judge is effectively the kind of you know I don’t know if it’s politically acceptable but the kind of lunatics running the asylum” — Samuel Colvin
Automated Prompt Optimization with Jepp(ar)
Jepp(ar) keeps the best-performing prompts on a Pareto frontier, breeds them, and proposes new candidates. Colvin wired it to a Pydantic AI proposer agent that reads the failure context and writes a new system prompt, which then runs through the same eval loop. He ran the Jepp(ar) optimization run hitting 96.7% live, spending roughly $2 on 400 calls.
The result was 96.7% accuracy, up from 92% for the hand-written expert prompt. The optimized prompt is verbose and machine-generated. One audience member caught a side effect: it excluded aunts and uncles, a classic overfit to the training split.
When Optimization Pays Off
Colvin’s answer on when optimization pays: it matters most on private data the model has never seen. A bank’s internal operating spec, a PE firm’s 200 million invoices — context the model can’t guess from training. His clearest number came from Shopify:
“they switched from basically just giving the entire website to GPT5 and saying what is this to using an agent and using a Quen model and Jeeper to optimize the prompt. and they got the price down from $5 million a year to 60 60 $73,000 a year” — Samuel Colvin
For generic public tasks, frontier models get it right without optimization. The hard sell is demos: if the example data is public, the model already knows it.
Managed Variables: Changing Agent Behavior Without Redeploying
Logfire Managed Variables store agent config (prompt, model, temperature) as a typed Pydantic model, then serve it to the running process on each request. No redeploy needed. Colvin ran the Managed Variables live language swap demo: he edited the prompt in the Logfire UI to say “reply in French,” dialed targeting to 100%, and the next response came back in French, then German. The targeting slider provides built-in A/B testing.
The intended end state: given an eval set, Logfire runs optimization in the background and hill-climbs toward the best prompt without human intervention.
Q&A
Does Jepp(ar) produce prompts that only work with the model you trained on? Yes; the optimized prompt is model-specific and you would need to re-run optimization if you change models, which is one reason most teams skip evals entirely. ▶ 52:06
Isn’t this competing with fine-tuning as a strategy? They are competing strategies; fine-tuning costs tens of thousands of dollars and is often superseded by the next model, but it applies in finance and other high-volume private-data cases. ▶ 56:07
How do you handle variance in eval results? Run each case many times; some hedge funds spend $20,000 a night running their full eval suite, and if you own GPUs the cost is effectively zero so you run everything 100 times. ▶ 49:06
What’s your personal take on whether Jepp(ar)-style optimizers work in practice? They make the most sense when pushing a cheaper or faster model to match a frontier model’s performance on private data the model has never seen. ▶ 41:40
Notable Quotes
generally defining your own is far better than LLM as a judge because the LLM as a judge is effectively the kind of you know I don’t know if it’s politically acceptable but the kind of lunatics running the asylum Samuel Colvin · ▶ 18:32
I know a bunch of hedge funds who are spending about $20,000 a night running their whole set of evals Samuel Colvin · ▶ 49:31
they switched from basically just giving the entire website to GPT5 and saying what is this to using an agent and using a Quen model and Jeeper to optimize the prompt. and they got the price down from $5 million a year to 60 60 $73,000 a year Samuel Colvin · ▶ 30:26
Key Takeaways
- Jepp(ar)’s genetic loop pushed a Pydantic AI agent from 87% to 96.7% accuracy on political-relation extraction by proposing and testing candidate prompts.
- Deterministic evals against a golden dataset beat LLM-as-judge; optimization matters most when models haven’t seen your private training data.
- Logfire Managed Variables let you swap a production agent’s prompt or model without redeploying, enabling live A/B testing of optimized prompts.