94% of MCP Servers Fail Compliance

▶ Watch (4:23)

Sterling Dreyer built MCP Debugger to run live tests against MCP servers. He tested hundreds of servers from registries and GitHub repos. Only 6% passed every check. 85% failed two or more. Many servers skip origin header validation. Others omit WWW-Authenticate headers. These failures create security risks and force clients into fallback paths. Dreyer says most users won’t notice, but the non-compliance is widespread.

The Spec Moves Fast, Tooling Lags Behind

▶ Watch (5:55)

The MCP spec has only four releases, but each one introduces major changes. Some suggestions became hard requirements. Dreyer struggled to find testing tools. He discovered the official conformance test suite only this week. It has 50 stars on GitHub. Client SDKs have tens of thousands. He built MCP Debugger and Tool Bench to fill the gap. Tool Bench statically analyzes server code for spec compliance. Dreyer wants conformance tests run in CI/CD for every server.

API Keys vs OAuth: A Scale Problem

▶ Watch (10:20)

API keys are not part of the MCP spec. Yet many servers require them. Users must fetch keys, often without granular permissions. OAuth was added in V2 and is more secure and user-friendly. But small developers find OAuth too heavy to implement. Dreyer sees a scale problem: large companies can add OAuth easily, but hobbyists cannot. The result is continued API key usage and lower security. Frameworks should handle OAuth complexity so users do not have to.

Solutions: Framework Ownership and Registry Gatekeeping

▶ Watch (13:00)

Dreyer argues frameworks must enforce compliance. Tasks like origin header validation and WWW-Authenticate headers should be automatic. Users should not have to remember them. He also suggests registry gatekeeping: servers that fail badly should not appear in official lists. Client-side rejection is another idea. Clients could warn users about non-compliant servers, like browsers warn on bad certificates. This requires spec-level mandates and client support from tools like Cursor.

Q&A

Did the compliance problems lead to building Arcade? No, they were discovered during Arcade’s development. ▶ Watch (16:39)

What tools can developers use to check compliance? MCP Debugger provides a detailed checklist; the official conformance test suite needs more work. ▶ Watch (17:35)

Notable Quotes

94% of servers we tested failed some kind of test. So, only 6% fully complied. Sterling Dreyer · ▶ Watch (4:23)

API keys are not in the spec at all. Sterling Dreyer · ▶ Watch (10:20)

It should be something that everybody should be running in CICD. Sterling Dreyer · ▶ Watch (8:47)

Key Takeaways

  • 94% of MCP servers fail at least one compliance check.
  • Common failures include missing origin headers and improper input schemas.
  • MCP needs better conformance tools and framework-level enforcement.

About the Speaker(s)

Sterling Dreyer is a founding engineer at Arcade.dev, where he works on backend, infrastructure, and MCP integration. Before Arcade, he was the second engineer at Featureform, a feature store platform acquired by Redis.