AI
Automation and Incident Response Integration
Jeff focused on Slack chat ops for real-time vulnerability tracking. Bots can auto-assign severity and kick off tracking into any system. Garrett’s team built a custom application on ServiceNow because no existing tool met their needs. They integrated a two-way API with their bounty vendor, plus patch-level monitoring. That integration lets them check patch adoption rates across the customer base. The app also stores root cause notes, exploit samples, and draft disclosure communications.
Building a PSIRT from Scratch
Garrett joined ServiceNow in 2022 with no PSIRT team or formal program. He used First.org’s services framework to define four phases: discovery, triage, remediation, disclosure and education. For complex incidents, they track multiple work streams — a main line issue plus variant hunting. The team runs post-incident reviews (response mechanism) and post-mortems (system failure). Both use a blameless approach: assume everyone made the best decision with the information they had.
Metrics and Storytelling
Garrett argued that metrics must justify the program’s cost — a million dollars plus per year. He established a CISO-level OKR tracking program health, board-visible every quarter. Qualitative signals included variety of applications hit, diversity of bug classes, and frequency of critical reports. He warned that flirting with a ratio of critical to low reports could show improving efficiency. Jeff added that trending vulnerabilities by root cause (e.g., seven XSS in one week) is a metric teams and leadership both understand.
Researcher Retention and Cultivation
Jeff emphasized non-monetary rewards. A swag store cost less than $50,000 over two years including shipping. Co-authoring blog posts and Hall of Fame acknowledgments build loyalty. He proposed patch snippet collaboration: showing a researcher the proposed fix and asking for bypasses. Garrett said he uses the researcher pool as a talent pool. Hiring a top researcher reduces ramp-up time, though it temporarily drops bounty volume. They also give beta feature access to top researchers with inflated bounties and direct developer contact.
Disclosure and Program Maturity
Disclosure needs a repeatable process must be repeatable — every researcher gets the same experience, playbook, and legal/comms alignment. Garrett distinguished product programs from enterprise programs: product scope may be one subdomain but massive behind the scenes. He argued program maturity does not require going public. Key measures are whether all desired assets are in scope, researcher base grows report validity, and the team can respond to high/critical reports consistently within internal SLAs. Jeff recommended stress testing every process change before scaling.
Q&A
How do you choose KPI/OKR metrics? Garrett suggested reading Measure What Matters and separating objectives from key results. He avoids simple volume graphs and pushes for ratios like critical-to-low severity. ▶ Watch (38:54)
How is automation from vendors like Expo changing the space? Jeff noted an increase in low-quality submissions but thinks vendors can handle tier-one triage. He wonders if the influx will dwindle once older vulnerabilities are burned down. ▶ Watch (41:10)
How do you handle global researchers and sanctions? Both use bug bounty vendors to manage payments, taxes, identity verification, and sanctions. ▶ Watch (42:08)
How do you set the right bounty amounts? Jeff recommends recalibrating every six months by comparing to peer programs in and out of your industry. Asking researchers for honest feedback is also effective. ▶ Watch (43:01)
Notable Quotes
“It’s also important to note that Book Bounty platforms offer integrations with many tools and like mixing this with a custom app really leverages like one the platform’s API and then it gives you like the best results in my personal opinion.” Jeff Guerra · 03:36
“if you want to know when each report comes in, that’s an easy integration, right? Jeff Guerra · 02:47
“the biggest reason to go down this road, um I think a lot of managers feel compelled to just have metrics like what are the numbers without thinking about why you’re really doing it.” Garrett McNamara · 18:05
“Standing up a swag store was less than $50,000 for like 2 years.” Jeff Guerra · 22:57
“Do you think we should do call?” “I don’t know. So what we did was we kind of actually naturally got there mostly. So we’re in four different countries, you know, a number of different continents.” Garrett McNamara · 16:28
Key Takeaways
- Start your bug bounty program small and stress test every process change.
- Use two-way API integrations with your bounty vendor and a custom app or Slack automation for efficiency. Build a PSIRT using First.org’s services framework as a maturity roadmap. Use CISO-level OKRs with board visibility to justify program costs. Keep researchers engaged with swag, co-authored content, and early access to beta features.
About the Speakers
Garrett McNamara is Senior Manager of Product Security Adversary Management at ServiceNow. He manages cyber risk as a core business discipline and leads global Adversary Management and Product Security programs spanning PSIRT, Product Vulnerability Management, and Mergers & Acquisitions.
Jeff Guerra is a Senior Product Security Engineer at GitHub. He enjoys bounties, application security, and vulnerability disclosure programs. He is an advocate for the effectiveness and community engagement that comes with vulnerability disclosure.