The Dual Audience Dilemma

▶ Watch (0:22)

Over 40% of Apollo’s documentation traffic now reads docs with vectors, not eyes. This includes AI user agents, AI crawlers, and direct tool usage. Humans want explanations, context, and narratives. Agents want patterns, strict do’s and don’ts, and code examples. Documentation strategies built over decades for skimming humans break down when the fastest-growing audience parses and pattern-matches instead.

First MCP Tools: Search Then Read

▶ Watch (3:04)

Apollo’s first MCP tools exposed existing platform capabilities: search docs and read docs. An agent would search for a query, read a page, then search again and read another page. A single answer could consume 25,000 tokens. This pattern wasted calls, tokens, and time. The core guideline: context is courtesy. Providing exactly what the agent requests in as few tokens as possible determines MCP server quality.

Building a Purpose-Built Research Tool

▶ Watch (6:03)

Abdelsamed reversed his approach. He started with the interface he wanted agents to use and built backward. The new research docs tool takes an array of questions and a context parameter. It returns exact chunks of documentation, not pages. An AI-powered chunking pipeline uses frontmatter instructions per page — changelog pages get one chunk per major version, API reference pages get one chunk per property. At build time, the pipeline generates questions that map to each chunk, then embeds the abstract, questions, and content into a 3,072-float vector array.

Testing MCP Servers with Agents and Statistics

▶ Watch (12:50)

To prove the new tool was better, Abdelsamed built an end-to-end testing framework. Two agents execute each test. The executor agent uses the MCP server to complete a task, mirroring the user’s environment with the same model and tools. A scoring agent with read-only access evaluates the output against a pre-written binary rubric. Switching from floating-point scores to binary true/false checks improved accuracy. Running tests back-to-back controls for model version changes. Statistical analysis using Fisher’s theorem confirmed the new tool required 70% fewer tool calls and 50% fewer tokens with the same output quality.

Notable Quotes

Over 40% of our traffic now reads docs with vectors and not eyes. Daniel Abdelsamed · ▶ Watch (1:24)

Context is courtesy. Daniel Abdelsamed · ▶ Watch (5:16)

This new tool took 70% less tool calls than previously and 50% less tool tokens with the exact same quality of the output. Daniel Abdelsamed · ▶ Watch (19:56)

Key Takeaways

  • Over 40% of Apollo’s docs traffic is now agentic, not human.
  • Purpose-built MCP tools beat exposing existing capabilities.
  • Test MCP servers with agents, binary rubrics, and statistical analysis.

About the Speaker(s)

Daniel Abdelsamed is a Staff Software Engineer at Apollo, where he has spent the last four years architecting and scaling the company’s documentation platforms. With nearly a decade of experience in TypeScript and application design, he focuses on building durable, developer-friendly…