Diagnosing a Kubernetes Typo in 2 Minutes

▶ Watch (0:02)

Contoso, a demo pet store, loses checkout. A user trying to buy catnip and cat toys cannot complete a purchase. The on-call engineer opens Headlamp, the Kubernetes UI maintained as a CNCF SIG UI subproject, and queries the AI assistant powered by Holmes GPT. The engineer suspects DNS. Holmes GPT runs kubectl and then fires an eBPF gadget from Inspector Gadget to capture a TCP dump. The dump shows the app is calling order-service, which does not exist. The correct service name is orders-service.

From Root Cause to Merged PR Without Leaving the UI

▶ Watch (2:00)

Holmes GPT confirms the typo by cross-referencing the TCP dump with the live service list. The engineer tells it to submit a pull request with a description suitable for incident management. The PR is merged immediately because the team is in hotfix mode. Palma notes the whole sequence, diagnosis through merge, took 2 minutes. Holmes GPT then drafts the root cause analysis, attributing the bug to a developer whose local environment used order-service while production used the plural form.

GitOps Sync Closes the Loop

▶ Watch (2:29)

The fix does not go directly to the production cluster. The team uses the Headlamp Flux plugin to sync from source. Flux picks up the merged PR and rolls the corrected configuration out. Palma walks through this step to show that the AI-assisted workflow respects existing GitOps gates. The PR-then-sync pattern keeps the speed gain without bypassing the production controls the team already has in place.

Operator Mode: Catching Problems Before Customers Do

▶ Watch (3:13)

Palma asks whether the team could do better than waiting for a user ticket. Yellin’s answer is Holmes GPT operator mode, recently launched by Robusta. In operator mode, Holmes monitors the entire cloud environment continuously, including on-premise virtual machines and network devices. It watches every deploy from coding agents shipping multiple PRs to production each day. When something breaks after rollout, Holmes catches it before customers notice and triggers the fix immediately.

Self-Mutating Agents on the Road Map

▶ Watch (4:21)

Yellin flags one road map item as the most significant: self-mutating agents. Holmes GPT already connects to any MCP server, any raw HTTP API without MCP, and any database through generic connectors. When customers ask for a new integration, the current process is that Robusta’s dev team opens a coding agent, builds it, and ships it. The self-mutating agent concept would let Holmes write its own integrations. Yellin cites a customer request to connect Holmes to Uber so it could dispatch taxis to on-call engineers.

Notable Quotes

it actually found that as well when listing the services. So, it’s telling me, “Hey, there’s a typo here. You’re pointing to the wrong service that doesn’t exist.” Jorge Palma · ▶ 01:55

With operator mode, you can ship at the speed of AI and also catch problems and fix them at the speed of AI. Natan Yellin · ▶ 04:12

And they say, “Well, so we can send a taxi to pick up our on-call engineers.” Natan Yellin · ▶ 04:56

Key Takeaways

  • Holmes GPT used an eBPF TCP dump to find a service name typo in under 2 minutes.
  • The Headlamp Flux plugin enforced GitOps promotion even during a hotfix.
  • Operator mode monitors deployments continuously and fires before customers file tickets.