AI
Credibility Budget Determines Program Success Before Tool Selection Does
Scott opens by asking: does your company have a project or product mindset? That question changes implementation timelines. Project-minded organizations fund security through one-time initiatives with heavy management overhead. Product-minded ones ask how to improve continuously. Before selecting a tool, security teams need to answer two questions: is the company checking boxes or committing to protection, and what are developers’ real priorities? Skipping those conversations is where credibility gets spent without a return.
Reporting on Findings Punishes the Wrong People
Most AppSec reporting leads with high and critical finding counts. Scott calls that the wrong metric. Teams that send weekly reports to the CIO and call out whoever has the most findings are spending credibility without building behavior change. Developers rank security as priority number six on average. Positive reinforcement drives more change than negative. The Shopify AppSec team called out the engineers who closed the most issues each week and used that as a recruiting signal.
Matching SCA Tool Type to Program Maturity
SCA programs grow through four stages: visibility, workflow, focus, and governance. Scott maps three tool types to that progression. Manifest file scanners give quick visibility but leave gaps in ecosystems that don’t enforce lock files, Maven and older .NET being the main examples. Buildtime scanners take longer to install but reduce false positives through reachability analysis. Runtime tools deliver the best prioritization because they report only what executes in production, but they require solid test coverage and add performance overhead.
Three Hidden Costs in Every SCA Tool Choice
Scott names three hidden costs: relevance, prioritization, and accuracy. Relevance is paid in time, either installing runtime agents on servers you already know matter, or curating a software inventory so manifest scanners know which repos to ignore. A CVE flagged in a migration script dormant for four years is a real example he’s seen result in deletion rather than remediation. Prioritization is paid in resources; buildtime scans need more CPU than standard GitHub runners supply. Accuracy is paid in complexity; manifest scanners skip dependency resolution.
Q&A
How do you measure what’s going well without just tracking everything that isn’t bad? Report on Jira tickets closed, not open findings. Highlight teams that built reusable pipeline patterns and centralize those as reference architecture. The goal is behavior change, not a metric. ▶ 30:05
Should security teams gate builds on SCA findings? Gate on net new risk, not existing technical debt. Breaking a build for a CVE that was already present stops 60 engineers for work one person will fix. Start with one small innovator team, gate on direct reachable dependencies, then expand. ▶ 33:20
What if the upstream maintainer hasn’t fixed the vulnerability? Some teams write policy that the SLA clock starts when the upstream fix ships. Some open source vulnerabilities are never fixed. Scott’s team has investigated reported CVEs only to find the vulnerability does not exist. ▶ 36:04
Notable Quotes
I have one throat to choke. Jamie Scott · ▶ 6:18
Now, relevance is paid for with time. Jamie Scott · ▶ 21:37
Perfect becomes the enemy of better Jamie Scott · ▶ 28:01
Key Takeaways
- Start AppSec conversations with people and process questions, not tool selection.
- Report on what teams fix and celebrate it; stop reporting raw finding counts.
- SCA tool type should match your current maturity stage, not your aspirational one.
- Separate existing technical debt from net new risk before setting any security gate policy.
About the Speaker(s)
Jamie Scott, CISSP, CCSP, is the founding product manager at Endor Labs, where he builds the next generation of dependency management solutions. Before Endor, he held product manager roles at Redis and StackRox. Red Hat acquired StackRox in February 2021.