AI
The 3 AM Problem: Why Scattered Context Slows Recovery
Liftoff processes billions of bid requests daily. When something breaks at 3 AM, the on-call engineer opens a laptop, checks dashboards, digs through Slack for prior context, and queries systems before understanding the problem well enough to fix anything. Documentation sits across Grafana, Confluence, Slack, Kubernetes, S3, and Google Cloud. Each verification step burns time that directly extends mean time to recovery. A GCP outage in October 2025 lasted hours and cascaded across multiple services at once, making a bad situation worse. Running active-active infrastructure to prevent that costs 50x to 70x more.
MCP as a Standard Interface for Production Systems
MCP is a standard interface between an AI agent and the systems it needs to reach. MCP clients include ChatGPT, Claude, Cursor, and Claude Code. MCP servers include Grafana and GitHub. The protocol is JSON-RPC over STDIO or SSE. A client connects, discovers what the server exposes, and that is the entire handshake. No custom SDK per integration. Liftoff ran a Trino MCP pilot from August 2025 to March 2026. Sixty-one percent of adopters were non-engineers. Sixty-five percent of all users became daily users. Average MCP familiarity at sign-up was 2.15 out of 5, yet adoption held.
OAuth Proxy and Agent Skills: The Foundation Layer
Connecting agents to production databases and cloud infrastructure without authentication hands unauthenticated access to the entire stack. Liftoff built an open-source library called Liftoff MCP Proxy. It adds OAuth 2.1 authentication to any Go MCP server with a single middleware call. Token validation, error handling, and SSO support for Okta, Google, and Azure AD are included. Users sign in once with the same SSO flow they use everywhere else. No new passwords, no API keys. On top of that, Liftoff introduced agent skills: markdown files encoding team knowledge. Skills cut data query iteration time by 93% and infrastructure cost tracking by 95%.
Combining CLI Tools and MCP for the Right Situations
CLI tools are fast and produce predictable output for engineers who already know them. MCP adds authentication, guardrails, and a guided experience for users who will never configure a local setup. MCP is what made adoption work for the 61% of non-engineers. Liftoff combined both: agents get the same kubectl, GitHub CLI, and cloud CLIs that engineers already trust, plus MCP servers for Grafana and PagerDuty. A skill can instruct the agent to check a Grafana dashboard for a service. Whether that resolves through MCP or CLI does not matter, as long as the job gets done.
Calypso: Automated Incident Investigation at 87-93% Accuracy
Calypso is Liftoff’s incident response agent. When an alert fires, a routing layer filters noise and duplicates, then triggers the agent if severity warrants it. The agent investigates using skills to select the right runbook, MCP and CLI tools to query Kubernetes clusters, AWS Route 53, and other systems, and a framework layer to consolidate findings. Results post directly to a Slack thread. The on-call engineer reviews the root cause analysis and decides whether to act or hand off to a second agent. Early results across several months show 87% to 93% success rate across all investigation types, with 58% of total alerts currently covered.
LLM-Orchestrated Disaster Recovery: 87% Pass Rate vs. 35% Industry Average
Calypso reacts to incidents. Liftoff also built a proactive prototype called LLM-Orchestrated Disaster Recovery. A scheduled trigger, daily, weekly, or monthly, spins up parallel service provisioning, runs validation checks, generates a report for system owners, then tears everything down. Infrastructure-as-code was translated into agent skills so the agent follows the original system structure closely. The pipeline runs fully automated with no human watching. Early pass rate is 87%. Companies that run disaster recovery manually once or twice a year average 35%. A side effect: frequent rehearsal improves the provisioning framework itself, cutting turnaround time when spinning up new services.
Notable Quotes
documentation and information are scattered Grafana confluence slack trade in everywhere in system in cubities in 53 Google cloud is everywhere Tommy Nguyen · ▶ 02:05
the cost to run the active active infrastructure costs us like 50x or or 70x more. Tommy Nguyen · ▶ 04:26
people don’t need to understand the protocol to get the value from it. Tommy Nguyen · ▶ 08:32
Nobody was asked nobody was assigned to contribute and nobody was told that they had to. Tommy Nguyen · ▶ 17:10
We get around 87% to 93% success rate across all investigation types. Tommy Nguyen · ▶ 23:30
Key Takeaways
- Liftoff’s OAuth 2.1 MCP proxy reduced auth friction to a single SSO sign-in for all tools.
- 61% of Trino MCP adopters were non-engineers; 65% became daily users within months.
- The automated disaster recovery pipeline passes at 87%, more than double the 35% manual industry rate.
About the Speaker(s)
Tommy Nguyen is an Engineering Manager at Liftoff Mobile with expertise in cloud-native and AI-driven systems. He specializes in Kubernetes, scalable architectures, and AI integration via Model Context Protocol. He was an early adopter of MCP at Liftoff, building the authentication and skills infrastructure that expanded access to production data for engineers and non-engineers alike.