AI
What Holmes GPT Does in Production
Holmes GPT is an open-source AI agent connected to Prometheus, Loki, Datadog, New Relic, and cloud providers. When an incident occurs, an engineer asks Holmes why a pod is crashing or why latency spiked. Holmes queries the connected data sources and returns a root cause analysis within minutes, including a recommended fix. Arik Alon, CTO of Robusta, maintains Holmes and submitted it to the CNCF together with Microsoft, which uses it inside the Azure CLI.
A 70% Metric Drop Caught and Fixed Without an On-Call Spiral
A team merged a large PR, likely written by a coding agent. The bidding rate dropped 70%, cache hit ratio fell, and latency spiked. Holmes caught all three signals and sent a Slack notification naming the deployment version (241) as the cause. It dug into the codebase and found the TTL cache had been set to zero, disabling caching entirely. Holmes then opened a GitHub PR with a fix. Arik’s only remaining task was reviewing and merging it.
Health Checks Written in Plain Language
The workflow starts with a Holmes health check, defined in plain English against a specific deployment. Arik’s check told Holmes to verify bidder deployment health, scan logs for errors, check latency metrics, and flag any degradation above 10%. If an issue appeared, Holmes was instructed to attempt a GitHub fix or, failing that, open a rollback PR. No custom scripts or alert routing rules are needed. The health check reads like an on-call runbook written in a sentence.
The Holmes Operator: From Health Check to Pull Request
The Holmes operator picks up health check definitions and executes them. In the bidding rate incident, it queried Prometheus to detect metric degradation, checked Kubernetes pod logs, then moved to GitHub to identify the offending code change. The operator opened the fix PR without human prompting. Beyond GitHub PRs, Holmes can trigger GitHub Actions, run kubectl commands to restart or scale pods, or call any other remediation handler. The PR example keeps a human in the loop; other handlers can act autonomously.
Notable Quotes
Holmes is telling me there was a deployment of deployment of version 241 and there is some critical issue. There is a drop in the metrics of 70%. Arik Alon · ▶ 02:20
someone basically set the TTL cache to zero which basically disabled the cache. Arik Alon · ▶ 02:45
Holmes can be can integrate it with any remediation handler. Arik Alon · ▶ 04:21
Key Takeaways
- Holmes detected a 70% bidding rate drop and traced it to a zero TTL cache setting.
- Health checks are written in plain English and map to specific Kubernetes deployments.
- The Holmes operator queries Prometheus, pod logs, and GitHub without human prompting.
About the Speaker(s)
Arik Alon is the CTO and co-founder of Robusta, an open-source platform focused on cloud observability and incident response. He maintains Holmes GPT and led the project’s submission to the CNCF alongside Microsoft.