AI
How Every Agent Works: the Agentic Loop
An agent accepts a user’s intent, combines it with a system context describing who the agent is and what it can do, then sends both to an LLM. The LLM either answers directly or requests a tool call. The agent executes the tool and feeds the result back. That loop repeats until the LLM returns a final answer. LLMs already know everything public. What they lack is company knowledge: policies, runbooks, and per-team conventions that separate a new hire from someone who knows how things are done.
Live Demo: Diagnosing a CrashLoopBackOff with kubectl Tools
The audience voted LangGraph as the agent framework. Whitney then gave the agent three tools: kubectl get, kubectl describe, and kubectl logs. Acting as a panicked developer with no cluster access, she asked the agent to figure out why her application was broken. The agent spotted a CrashLoopBackOff, described the pod, pulled logs, and returned a diagnosis. It then tried to suggest a fix using Qdrant, which was wrong. The demo exposed the core gap: with only kubectl tools, the agent could list everything but had no way to know which of hundreds of CRDs was relevant.
Semantic Search: Finding the Right Database in a Crowded Cluster
A controller running in the cluster synced all relevant Kubernetes resources into a Chroma vector database (the audience’s second vote). Embeddings convert text into roughly 1,000-dimensional number vectors. Proximity search against those numbers is what lets the agent skip listing every CRD and jump straight to what matters. Given the query “which database is meant for my application,” the agent queried the capabilities collection, identified managed Crossplane services, asked Whitney which team she was on, and returned the exact database resource provisioned for the You Choose project, including configuration and deployment instructions.
Guardrails: What the Agent Can and Cannot Apply
Viktor added a kubectl apply tool to the agent after granting permission. Whitney immediately asked it to deploy a Tron game. The agent tried, found Tron absent from the platform-approved custom resources list, and refused. It then correctly deployed the Crossplane database because that resource type was approved. The lesson: guardrails must be deterministic. Informing the agent which tools are off-limits at the tool level produces shorter loops than letting it hit RBAC failures repeatedly. Kyverno policies and RBAC rules loaded into the vector database let the agent know its own permissions before it tries anything.
Observability: Tracing Agent Behavior with OpenTelemetry
Agent inputs and outputs are unbounded. A user can type anything; the LLM can do anything within its tools. That makes traditional input/output validation useless. Traces are the practical substitute. The talk used OpenTelemetry with GenAI semantic conventions, storing traces in Jaeger (the audience’s third vote, over Datadog, to Whitney’s mock horror). In Jaeger, each span showed which tool was called, in which order, with what result. Cost appears in the trace metadata. Semantic conventions matter because backend tooling and platform teams share a consistent vocabulary for what each span means.
Q&A
How should guardrails be implemented, and how do you stop the LLM from going around them? Guardrails should be deterministic wherever possible; combine tool-level restrictions scoped to specific namespaces with cluster-level RBAC and Kyverno, and load those policies into the vector database so the agent knows its permissions before acting. ▶ 31:07
When does a vector database beat just giving all knowledge directly to the agent or sub-agents? When the knowledge base is large, sending everything to the LLM context is too slow and too expensive; Slack history alone can exhaust a context window before the LLM finds the relevant message. ▶ 34:05
How do you measure the quality of individual tool or skill calls? Use an evals framework with AI-validated outputs, track thumbs-up/thumbs-down user feedback, count turns to completion, and watch failure rates such as repeated misinterpretation of the same data. ▶ 35:28
Notable Quotes
guardrails should be deterministic whenever possible and then you can also use kivero or any of your normal tools to on your agents is now you give your agent an identity and then you create kyverno rules about what that identity can or cannot do. Viktor Farcic · ▶ 31:30
you cannot assume that LLM will respect you just as it will. So kind of you know intelligence, no respect for intelligence. Viktor Farcic · ▶ 33:54
it can take hours just to go through all the slack messages and it literally needs to go through all of them to find out and long before it goes through all of them it will say I’m out of context. Viktor Farcic · ▶ 34:49
somewhere it has to be deterministic that’s the big takeaway Viktor Farcic · ▶ 32:56
Key Takeaways
- Semantic search over a vector database lets an agent find relevant cluster resources without iterating every CRD.
- Guardrails must be deterministic at the tool level; RBAC and Kyverno add a second enforcement layer.
- OpenTelemetry with GenAI semantic conventions is the only consistent way to trace what an agent actually did.
About the Speaker(s)
Viktor Farcic is a lead rapscallion at Upbound, a CNCF Ambassador, Google Developer Expert, CDF Ambassador, and GitHub Star. He is a published author, hosts the DevOps Toolkit YouTube channel, and co-hosts the DevOps Paradox podcast.
Whitney Lee is a Senior Technical Advocate at Datadog. She builds original frameworks that connect observability, AI, and platform engineering, and brings clarity to complex cloud native technologies through her work and co-hosted shows.