AI
Staying Ahead of a Fast-Moving Technology Wave
Liberty Mutual’s emerging technology team had been researching GenAI since 2022 and rated it “worthy of further investigation.” Then ChatGPT launched the week after Thanksgiving. The existing research gave them something to bring to leadership immediately, and a pre-existing Microsoft partnership meant they could get model access and build an internal platform fast enough to matter.
The lesson: treat technology research as continuous maintenance. Keeping work on the shelf — even on technologies that haven’t won yet — builds the foundation to move quickly when one suddenly does.
Shipping a Tool Is Not the Same as Adoption
Liberty IT got GitHub Copilot to 90% adoption within a year and Liberty GPT to 70% of Liberty Mutual’s global workforce. Surveys still showed developers stuck on basic completions, ignoring advanced features. The target kept shifting: completions to agent mode to an async coding agent inside a single year.
“just because you’ve shipped it or you’ve got it out there doesn’t mean it’s actually being used effectively” — Stuart Greenlees
Liberty IT responded with a formal six-stream GenAI learning mission: universal AI literacy, separate tracks for architects and people leaders, and monthly open office hours where senior architects demo current work.
Model Selection and Cost Control
Liberty Mutual runs 28 models internally, matched by use case. Gemini simplified domain terminology that users needed preserved; OpenAI kept the jargon intact. Cost differences compound fast: GPT-4.1 runs $8 per million tokens; Claude Opus 4 runs $75. One use case burned $20,000 in a few hours, shown in the $20k token burn incident (33:01). Switching that team to a cheaper model saved an estimated $40,000.
Match models to the job. Build cost dashboards, automate spend alerts, and re-evaluate as new models undercut existing prices.
Scaling GenAI Needs a Centralized Platform
Self-service model access — the approach that worked for cloud — collapsed under gated quotas, regional availability gaps, legal reviews, and security concerns about prompt injection. Liberty Mutual built a centralized platform with one API layer, quota allocated by use case, and consistent guardrails for the responsible AI committee. The GenAI northstar architecture that justified the investment took 12 architects months of debate to produce; they time-boxed it, shipped a “good enough” version, and iterated. Bain & Co. rated the second version top-quartile.
Engineer autonomy remains the goal. It waits for the governance framework to catch up first.
The Short Half-Life of GenAI Solutions
GPT-3.5’s token limit forced Liberty IT to build a summarizer-of-summarizers. Larger context windows made it obsolete. Hand-rolled instruction file templates were rendered redundant when GitHub shipped a Copilot customization page with better examples. Stuart called the pile of retired services a “tech graveyard”: summarizers, a two-way SMS integration, valuation frameworks, and the internal AI platform with an open grave next to it.
“the halflife of your genai solutions is very short” — Garth Gilmour
Build a composite architecture that makes services easy to drop, and focus effort on use cases only your company can do well.
RAG Is Harder Than the Tutorials Suggest
Off-the-shelf RAG libraries top out around 65% quality.
“off-the-shelf libraries maybe take you to about 65%” — Garth Gilmour
The remaining gap requires decisions on chunking strategy, embedding model, re-ranking, and custom evaluation metrics beyond the four standard ones most libraries ship. Groundedness — tracing each answer back to a cited source rather than a hallucination — is its own discipline. Liberty IT also learned latency the hard way: one GenAI feature defaulted to 15-second response times and users reverted to the old app. Optimizing the vector database, adding GPU compute, and applying progressive UI updates brought that down to 3 seconds.
Notable Quotes
just because you’ve shipped it or you’ve got it out there doesn’t mean it’s actually being used effectively Stuart Greenlees · ▶ 15:41
the halflife of your genai solutions is very short Garth Gilmour · ▶ 27:17
off-the-shelf libraries maybe take you to about 65% Garth Gilmour · ▶ 36:24
Key Takeaways
- Getting GenAI tools deployed is a starting point; structured learning programs are required to move from basic use to fluent, effective use.
- Supporting 28 models and building cost dashboards saved Liberty Mutual from runaway spend when one use case burned $20k in hours.
- Production RAG demands groundedness checks, citations, and an automated evaluation pipeline — off-the-shelf libraries top out around 65% quality.