AI
Vendor Selection Through Peer Networks
Richard Stennon of IT Harvest described the pattern: evaluate three vendors, pick the best demo, sign a three-year contract, then rip it out 18 months later. Mike Johnson’s rule is never sign a three-year deal on first engagement. He starts with one-year deals and builds short lists from peer CISOs, not Google or Gartner. Sara Madden inherited three-year licenses building a green-field program. She now pushes vendors to one-year or two-year renewals and pressures them on software quality across nearly every vendor in her tool set.
AI Output Requires Skepticism
Howard Holton of Gigome shared a Gartner stat: 74% of organizations see AI productivity gains, but only 11% see clear ROI. AI delivers polished formatting and authoritative tone that masks low quality. Sara Madden’s team uses AI daily in sec ops for code analysis but never trusts its initial output. They treat AI findings like false positives from any vulnerability scanner. Mike Johnson noted the shift from blind belief to double-checking over the past six months and pointed to AWS’s recent AI-induced outage as a reminder to verify.
People Risk Exceeds Technical Risk
Craig George of Guidepoint Security submitted a “What’s Worse” scenario: a security program held together by two exhausted heroes, or years of unmanaged service accounts and API tokens. Both panelists chose the people problem. Mike Johnson argued technical problems are far more solvable than people problems. Sara Madden agreed: lose your people and you cannot fix anything. The audience voted overwhelmingly the same way.
Tabletop Exercises Need Real Stakes
Joshua Copelan of Cresendo argues that tabletop exercises lie because the incentives are wrong. Nobody protects themselves. No real incident unfolds cleanly. Sara Madden runs quarterly tabletops across three functional teams. Her teams failed repeatedly but improved over two years. Mike Johnson brought in an outside firm to raise the stakes. The unfamiliar facilitator removes comfortable assumptions. The money spent on an external firm also creates pressure to extract value from the exercise.
AI Adoption Metrics Must Be Purposeful
Jensen Wang, CEO of Nvidia, said a $500,000 engineer should burn at least $250,000 in AI code tokens. Mike Johnson noted Wang’s financial interest in token consumption but agreed engineers should augment themselves with AI. Sara Madden measures AI effectiveness by business goals, not token volume. Her compliance team built AI bots to automate audit responses, displacing vendor solutions. A TPM on Mike’s team asked for a Cursor license despite never writing a line of code before.
Q&A
Would a $500,000 engineer who does not use $250,000 in AI code tokens alarm you? Mike Johnson noted Jensen Wang’s financial interest but agreed engineers should lean into AI augmentation. Sara Madden said metrics should tie to business goals, not token consumption. ▶ 39:36
What changed with agentic AI from last year to this year? Mike Johnson said the shift from standalone agents to smaller agents controlled by a master agent is the defining change. ▶ 42:55
Do you see yourself phasing vendors out because of AI? Sara Madden already made that call on a vendor a couple weeks ago. Mike Johnson said any vendor not differentiated from a generic LLM is replaceable. ▶ 43:45
What cool AI use cases have your teams built? Mike Johnson highlighted a TPM who asked for a Cursor license despite never writing code. Sara Madden’s compliance team built AI bots to automate audit responses. ▶ 44:32
Notable Quotes
Well, the first step is don’t sign a three-year deal the first time that you’re working with a vendor. Period. Mike Johnson · ▶ 7:06
We don’t trust it and we use AI every day. Sara Madden · ▶ 13:06
Your incident response tabletop, your IR tabletop, is lying to you. Not because the scenario is wrong, because the incentives are. David Spark reading Joshua Copelan · ▶ 32:48
We do tabletops quarterly, and my feedback 100% of the time is, you didn’t bring me in early enough. Sara Madden · ▶ 34:05
every time we renew a software license we ask ourselves can we replace this with AI? And I already made that call in a vendor a couple weeks ago. Sara Madden · ▶ 43:45
Key Takeaways
- Start vendor relationships with one-year contracts, not three-year deals.
- Treat AI output with the same skepticism as vulnerability scanner false positives.
- Quarterly tabletops with outside facilitators reveal fault lines internal teams miss.
About the Speaker(s)
David Spark is the producer of the CISO Series, a media channel of blogs, podcasts, and videos all on the cybersecurity ecosystem.
Mike Johnson is the CISO of electric vehicle manufacturer Rivian. He joined Rivian from Fastly where he was CISO for over 3 years. His cybersecurity career spans more than 25 years.
Sara Madden is the Chief Information Security Officer (CISO) of Convera, responsible for global cyber risk and compliance, cybersecurity operations, identity and access management. She has over 25 years of experience in cybersecurity.