AI
From Single-Shot Search to Agentic Retrieval
Two years ago, RAG was assembled from whatever was at hand: a language model, a search stack, and 500-token chunks. Keyword and vector search got the industry through its first wave of applications. But a single top-K query cannot handle a question spanning multiple products, containing typos, and referencing earlier conversation turns.
“it was built out of the tools available in the room” — Pablo Castro
Azure AI Search is now shipping agentic retrieval, a query-planning layer built into the service. Pass a slice of chat history to a knowledge agent. It decomposes the request, issues parallel sub-queries, reranks each leg, and returns a merged result ready to feed a language model.
How the Knowledge Agent API Works
To create a knowledge agent, you name it, point it at one or more indexes, set a relevance threshold, and pick the model to use for query planning. After that, you never write a search query again. The knowledge agent live demo (12:38) shows what happens when a user asks “which one fits more people?” after a prior turn mentioned TrailMaster and SkyView tents. The system recognized the comparison, branched into two sub-queries (“TrailMaster 10 maximum capacity” and “SkyView tent maximum capacity”), ran them in parallel, and merged the ranked results. The response includes the grounding text, source citations, and an activity log showing exactly how the query was planned.
Benchmark Results for Complex Queries
On internal benchmarks spanning multiple industries and languages (the MIML dataset), agentic retrieval produced a 40% increase in answer relevance and a 30% increase in result rate on complex queries. Complex is defined precisely: any question that cannot be answered by retrieving a single document. On simple single-topic questions the system holds even, never regressing. Groundedness scores, the measure of whether answers stay tied to retrieved content, remained flat across both query types. Castro flagged that the higher-level knowledge agent abstraction also gives the team room to improve internals without changing the API contract.
Multimodal Ingestion: Images, Layout, and Logic Apps
Real business documents have diagrams, arrows, and layout that carry meaning plain text does not. Three new ingestion options address this: image verbalization (each image goes to GPT-4o, which writes a text description that gets indexed), AI Document Intelligence layout extraction (pulls table structure and individual image assets so each is separately addressable), and Logic Apps connectors (pull from OneDrive, SharePoint, or hundreds of other sources without custom pipeline code). The multimodal ingestion wizard (26:33) shows the full flow from blob storage through verbalization to a working multimodal chat app in under five minutes.
Entra-Native Document Access Control
Smart retrieval finds everything, which makes access control non-negotiable. Manually propagating ACLs, expanding group memberships, and writing per-query filters is error-prone work.
“doing this manually is complex and it’s error-prone” — Pablo Castro
Azure AI Search now supports native Entra-based document-level permissions. Add three fields to an index (users, groups, roles), enable permission filters, pass a user token at query time, and the index returns only the documents that user can see. Group updates in Entra propagate automatically. A second feature, in private preview, extends this to Purview sensitivity labels, including decryption during indexing. The Entra ACL demo (41:53) shows the access boundary enforced live.
Agent-Friendly APIs and the Azure MCP Server
The Azure MCP server exposes Azure Search primitives to any MCP-compatible client, including VS Code in agent mode. Two features make search indexes particularly useful here. First, index schema already describes which fields are searchable, filterable, or facetable. Second, top-level resources now carry description fields where you can document what the index contains and how it should be used. In the agent-built outdoor gear app (51:35), VS Code read the index schema and description via the MCP server, then generated a working Next.js search UI with facet navigation from one prompt.
“every pixel you see here, it was just that prompt” — Pablo Castro
Notable Quotes
it was built out of the tools available in the room Pablo Castro · ▶ 2:20
doing this manually is complex and it’s error-prone Pablo Castro · ▶ 40:26
every pixel you see here, it was just that prompt Pablo Castro · ▶ 55:34
Key Takeaways
- Agentic retrieval decomposes complex multi-part questions into parallel sub-queries, raising answer relevance by 40% on benchmarks.
- Image verbalization via GPT-4o and AI Document Intelligence layout extraction are now first-class options in the ingestion wizard.
- Native Entra ACL enforcement at the index level removes the need for manual post-filter logic when building enterprise copilots.