AI
The AI Security Vendor Evaluation Problem
A client calls Kovalsky asking for an AI security vendor recommendation. Before he can answer, he needs to know their use cases, risk tolerance, and whether they’re more worried about an incident or filing it with the SEC. The client says: just tell me who’s good.
“hold on. Like, that’s not enough information.” — Maxim Kovalsky
That gap is the problem. Vendors fill it with marketing copy: “swarms of autonomous agents,” “broadest and most comprehensive,” “99% efficiency at sub 30 millisecond latency.” Between December and the talk’s submission date, Kovalsky’s team had analyzed 62 vendors. Six weeks later they were at nearly 80, with four new entrants per week claiming AI security coverage.
The Broken AI Governance Loop
The typical AI security procurement cycle looks like this: a team gets six vendor data sheets, builds a risk register by hand-jamming OWASP or NIST into a spreadsheet, invites four vendors to demo, sends two a proof-of-concept invite. Then someone asks whether AWS Bedrock’s GuardRails already handles this. Then whether DLP covers it. Then whether open source is cheaper. Three months in, the team needs more time. A consulting firm gets called. The process restarts.
“in the meantime, the customer chatbot is shipped.” — Maxim Kovalsky
The AI system goes live before governance does. That’s the loop.
How the Vendor Research Agent Works
The agent runs as an agentic loop inside Claude: a research agent and a QC agent working in tandem, shaped by system prompts and skills files refined over months. Their job is to find evidence for every vendor claim. That evidence comes from GitHub repos, API documentation, and user forums. Based on what it finds, the agent assigns a confidence rating. A score of five out of five means it pulled code samples from a GitHub repo showing exactly how a capability is instrumented.
Mapped capabilities feed into a custom risk taxonomy synthesized from OWASP LLM Top 10, NIST AI RMF, and MITRE ATLAS. Once a GenAI system enters the wizard, the tool cross-references that taxonomy against confirmed vendor capabilities, factors in existing strategic investments, and produces implementation guidance for GRC, cloud engineering, and security architecture teams.
Live Demo: Vendor Cards and the Risk Taxonomy
The tool starts with a synthesized AI risk taxonomy. Each risk category, such as sensitive data in AI inputs, maps to a required capability (DLP and redaction), guidance for building it in-house, and a split vendor list by implementation layer: application-side SDKs versus infrastructure choke points. Kovalsky drilled into data disclosure and pulled up the Crowdstrike Falcon AI data-disclosure coverage (11:56), showing the agent’s evidence that Crowdstrike covers that risk at both the network layer and on endpoints. The output is agent-gathered, not marketing copy, though Kovalsky was direct: significant human QC still catches hallucinations before anything ships.
Live Demo: Risk Assessment Wizard for a GenAI System
The wizard ran live against Adjuster IQ, an insurance claim system built on AWS Bedrock. Each choice — internal confidential data, employees over the internet, autonomous action scope — shifted the inherent risk profile in real time (13:08). Architecture questions followed: RAG, direct API, chat interface, document analysis.
Existing investments in Crowdstrike and Zscaler covered 8 capabilities. The gap analysis produced 15 remaining items (17:27). Several governance gaps were tagged “build” — internal workflows, no purchase needed. The rest resolved to five vendors, with a printable remediation report including AWS Bedrock Guardrails configuration guidance.
Q&A
Platform consolidation vs. best-in-class: how does the tool handle a Swiss-army-knife vendor like Wiz? The tool recommends platform vendors when that preference is selected, but Kovalsky said even the largest platforms (PaloAlto, Crowdstrike) cover only about 50% of the most relevant AI security risks, and Wiz currently has a small handful of AI security capabilities. ▶ 20:01
How do you maintain currency in such a rapidly changing vendor landscape? A GitHub Actions workflow runs every evening for about 12 hours, cycling through vendors so each one gets re-researched at least once a month, with a separate workflow to detect new entrants and traditional vendors adding AI security capabilities. ▶ 21:27
Notable Quotes
hold on. Like, that’s not enough information. Maxim Kovalsky · ▶ 1:33
in the meantime, the customer chatbot is shipped. Maxim Kovalsky · ▶ 6:14
even those big platform plays are still going to give you about 50% of of the risks that are most relevant. Maxim Kovalsky · ▶ 20:50
there’s an agentic world workflow that kicks gets kicked off every evening. So overnight you know it’s about 12 hours every day. Maxim Kovalsky · ▶ 21:32
Key Takeaways
- Map vendor claims to a risk taxonomy before demos, not after—procurement loops that skip this step ship unprotected AI systems.
- Score vendor evidence at the source: GitHub repos and API docs reveal capability gaps that marketing sheets hide.
- Even the largest platform vendors cover only about 50% of AI security risks; plan for a multi-vendor shortlist from day one.