How the Classical RAG Pipeline Works

▶ Watch (03:14)

Azure AI Search has run a full hybrid retrieval stack for two to three years. A user question goes through query writing (LLM paraphrases), then keyword (BM25) and vector search run in parallel. Reciprocal rank fusion merges the two lists. Semantic ranker, the cross-encoder that powers Bing, re-scores the top 50 and normalizes each to a 0-to-4 relevance score. Setting a threshold of 1.5 drops irrelevant chunks before they reach the LLM. Pamela Fox demonstrated this in the basic RAG chat demo (05:04), with HR documents, clickable PDF citations, and a visible thought process.

Where Basic RAG Breaks Down

▶ Watch (19:41)

Single-query RAG struggles when a question contains multiple distinct information needs. “What’s the difference in costs for copays versus split pays?” maps to two separate lookups, but a single search string will find partial evidence for each. The same problem appears in multi-turn conversations: the last user message often omits context established earlier, so a query built only from that message misses relevant documents entirely. These failure modes motivated the new Agentic Retrieval feature in Azure AI Search, released at Microsoft Build 2025.

The Knowledge Agent and Query Planning

▶ Watch (23:35)

A knowledge agent in Azure AI Search takes a full conversation history, runs an LLM query-planning step (1,270 input tokens in the demo), and fans out parallel sub-queries across the standard hybrid+semantic-ranker stack. Results come back two ways: a references list sorted by semantic ranker score descending, and a response string with results interleaved across sub-queries in round-robin order. The team found the interleaved format produced better answer generation because the LLM saw evidence from all sub-queries, not just the highest-scored one. The response string is pre-formatted JSON the LLM can consume directly, removing ten-plus lines of manual context assembly.

Wiring Agentic Retrieval into Foundry Agent Service

▶ Watch (39:51)

To wire agentic retrieval into Azure AI Foundry Agent Service, wrap it in a function tool. The function pulls the last five messages from the agent thread, calls the knowledge agent with those messages, and returns the interleaved response string. The Foundry agent then calls that function when it needs grounding context, alongside any other tools (Bing grounding, file stores). Pamela Fox walked through the agentic retrieval end-to-end demo (37:51): a health-plan question produced cost comparisons and citations, and the activity log showed three parallel sub-queries with per-query timing.

Evaluation Results and Model Selection

▶ Watch (47:47)

Internal evaluations on complex multi-document queries showed a 40% lift in answer relevance and a 30% improvement in answer rate compared to standard RAG. Simpler queries showed no regression.

“If you have complex queries that require information from multiple documents, we see a 40% lift in answer relevance over regular RAG.” — Matt Gotteiner

Model choice affects sub-query count. GPT-4.0 Mini generates more sub-queries than GPT-4.1, which can improve recall but extends latency. Setting a maximum sub-query cap (50 documents re-ranked per query) keeps costs predictable.

Q&A

Can agentic retrieval work with Postgres or other data sources outside Azure AI Search? Currently Azure AI Search only; Gotteiner said Postgres and other sources are on the roadmap but no date. ▶ 53:07

Does agentic retrieval eliminate the need for separate query rewriting steps? Agentic retrieval handles paraphrasing in the planning step, but preparing conversation history (summarizing or windowing) is still the developer’s responsibility. ▶ 57:32

Will query-planner customization support domain-specific acronyms? Not yet; Gotteiner confirmed the only current customization hook is the conversation history itself, and suggested stuffing a synonym list into the system prompt as a workaround. ▶ 59:56

Notable Quotes

If you have complex queries that require information from multiple documents, we see a 40% lift in answer relevance over regular RAG. Matt Gotteiner · ▶ 48:01

agentic retrieval will use more tokens because you have more context. Matt Gotteiner · ▶ 57:07

the only way to customize the query planner is to use the conversation history. Matt Gotteiner · ▶ 1:00:43

Key Takeaways

  • Agentic retrieval splits complex questions into parallel sub-queries, improving answer relevance by 40% on multi-document queries.
  • The knowledge agent returns a ranked references list plus an activity log showing token costs, query count, and timing.
  • Mini models generate more sub-queries than larger models, which can improve recall but increases latency and token spend.