AI
The Good, Bad, and Ugly of AI in Cybersecurity
Defenders are outnumbered. Attackers use AI. Alert volumes grow. New vulnerabilities pile up faster than analysts can clear them. The question is not whether to use AI on the defense side, it is how to use it without getting burned by vendors sprinkling the word “AI” on everything and refusing to explain the back end.
The bad: token pricing is as opaque as early cloud billing, and “it just works” is not an acceptable answer for a security team. The ugly: employees blocked from ChatGPT are already using it on their phones. Lodge’s position is direct.
“So, that that’s my take. AI is a guide, not necessarily a full-on guard. We need human in the loop.” — Brennan Lodge
How RAG Works and Why Self-Hosting It Beats the Cloud APIs
RAG has three parts: a sentence embedding model that converts documents into vectors, a vector database that indexes them by context and meaning, and an LLM that retrieves the right chunk and answers your question. The model stays on your network. Your policy documents never leave.
Running it inside your own VPC also costs less. Lodge ran the cost benchmark (15:22) on five analysts asking 20 questions a day, 100,000 tokens total: $500 a month self-hosted versus $600 through the OpenAI API. The dollar gap is small. The IP exposure gap is not.
MCP as the Integration Layer for GRC Workflows
MCP is the routing layer between your LLM and internal data sources. The host holds your model. The client orchestrates requests. The server holds the data. MCP connects them without custom plumbing.
For a GRC query, that looks like this: a compliance analyst asks whether the company falls under DORA. MCP routes the query to a vector database loaded with internal architecture docs and the DORA regulation. RAG fetches the relevant chunks. The answer comes back, confirmed, because the company provides payment and lending services to EU customers. No manual cross-referencing. One query, one response.
RAG-Powered Gap Analysis for Compliance Frameworks
GRC teams drown in policy documents scattered across Jira, SharePoint, and file repos, then get handed a three-month deadline to pass ISO 27001. Lodge’s fix: pull those documents into a vector database, then run them through a fine-tuned RoBERTa model that classifies text against ISO 27001, NIST CSF, and SOC 2 controls. The model outputs confidence levels, not binary pass/fail, so auditors see where coverage is thin. New legislation without an attached framework, like New York’s Shield Act, gets mapped the same way: translate the statute into the nearest framework controls, then score existing policies against them in bulk.
Audit Caddy Live Demo
Audit Caddy (auditcaddy.com, free for individuals) runs two BERT models against uploaded documents and routes queries through MCP to a local vector database. Upload a business-continuity doc and the classifier maps it across all 62 SOC 2 controls, each tagged with a confidence level — DR policy auto-mapped to 62 SOC 2 controls (28:10). Ask “do we have a disaster recovery policy?” and MCP returns the document with a summary — MCP-powered document search (30:03). Spot a gap and the tool prompts for company context, then outputs a watermarked draft in seconds — AI-generated privacy communications policy (31:21).
“This is not the easy button. You still have to go in there and customize and put your email addresses and justify what was written in the policy and understand what’s written in the policy.” — Brennan Lodge
How to Actually Deploy AI in Your Security Program
Get your security teams involved before the contract is signed. Press vendors on logging: can they show you what the model is doing on the back end? Lock down the prompts and apply RBAC to the vector database so retrieved documents only reach the people cleared to see them.
Start with RAG. It keeps your data inside your own VPC, gives you control over what goes into the vector store, and runs on local models now that GPU costs are falling. Your competitors are already using AI. The question is whether they’re doing it the secure way.
Q&A
Are we creating a new layer of opacity with AI for GRC instead of actually calculating risk — and will managers use it as an excuse to cut compliance headcount? Lodge argued AI works best as a referee between auditor and company rep, replacing absolute compliance percentages with confidence tiers (high/medium/low) while keeping humans in the loop and logging every action for audit trails. ▶ 36:01
Can RAG and AI reduce the burden on GRC teams from answering customer due-diligence questionnaires? Lodge said yes, framing vendor questionnaires and third-party risk under the same RAG use case, and proposed a real-time regulatory feed that drops new rules into the vector database and runs continuous gap scans instead of annual audit cycles. ▶ 40:22
How do you get executives and decision-makers to actually care about cybersecurity complexity? Lodge said the only reliable path is P&L: show executives what they currently spend on bloated GRC tooling and overpriced auditors, then demonstrate that AI-driven automation cuts those costs and corrects the industry’s pricing, rather than leading with technical complexity. ▶ 43:43
Notable Quotes
I don’t think AI is a new threat vector. It’s very similar to like a SQL injection attack. It’s just a new technology we have to deal with and use. Brennan Lodge · ▶ 7:17
So, that that’s my take. AI is a guide, not necessarily a fullon guard. We need human in the loop. Brennan Lodge · ▶ 11:29
We need to trust it but verify it. Brennan Lodge · ▶ 10:18
This is not the easy button. You still have to go in there and customize and put your email addresses and justify what was written in the policy and understand what’s written in the policy. Brennan Lodge · ▶ 31:39
Key Takeaways
- Run RAG inside your own VPC — keep sensitive policy documents and IP off third-party AI APIs entirely.
- Treat AI-generated compliance outputs as a starting point, not a verdict — confidence levels beat percentages, and human review is non-negotiable.
- Press every AI vendor for logging, token cost transparency, and audit trails before signing; ‘it just works’ is not acceptable for security.
About the Speaker
Brennan Lodge
Brennan is a self-proclaimed data nerd striving to save the world with a little help from our machine friends. He has held cyber security, data scientist, and leadership roles at JP Morgan Chase, the Federal Reserve Bank of New York, Bloomberg, and Goldman Sachs. Brennan holds a masters’ degree in Business Analytics from New York University and participates in the data science community with his non-profit pro-bono work at DataKind.