AI
The Security Scaling Crisis
Code generation from AI tools has skyrocketed. Security teams face more code, more vulnerabilities, and the same headcount. Lyft merges hundreds of pull requests daily. SAST tools produce thousands of findings. Manual design reviews take days for 25-page specs. Hiring more people and buying more tools only work to a point. Prioritization becomes guesswork when tools don’t understand organizational context.
Automated PR Security Analysis
The PR analysis agent extracts all API routes from a pull request. It checks for authentication and authorization changes across multiple phases. Each phase is a simple LLM call. The system runs the same workflow through two models (Sonnet and GPT) with a 60/40 weight split. It only generates a finding when consensus exceeds 50%. Smart route filtering excludes health check endpoints and accepted risks. The agent found 30 true positives: 4 critical, 16 high, 10 medium. All were authentication or authorization issues like IDOR and horizontal privilege escalation.
SAST Triage with Code Context
SAST tools flag SQL injection on line 47 but cannot tell if it is exploitable. The triage agent uses Sourcegraph’s MCP server to search code across multiple microservices. It evaluates risk assessment, authentication analysis, and external exposure. The agent checks if an endpoint reaches customer data and whether security best practices were followed. It runs 24/7 with consistent methodology. Before AI, triage took 30 minutes to hours per finding. After AI, the agent produces structured results that human engineers verify.
AI-Powered AppSec Design Reviews
Lyft’s four-person appsec team could not manually review every design spec. The V0 system let engineers upload a PDF or Google Doc and ask questions about authentication, authorization, and data handling. Reviews that took days now take 5 minutes. Every spec gets reviewed. The V1 agent downloads the document, analyzes architecture diagrams, performs threat modeling, and runs GDPR analysis. It synthesizes findings into a structured report. The agent finds issues at the design stage that were previously missed.
Key Lessons from Building AI Agents
Running the same workflow through two different LLMs and comparing outputs reduces non-determinism. Structured JSON output with hooks ensures parsable results. Skills let security engineers codify their triage process for any coding agent. Iterating on real vulnerabilities refines prompts through a feedback loop. Managing false positives preserves trust with engineering teams. Smart route filtering and threat modeling reduce token costs. Context engineering is the secret to finding and fixing vulnerabilities.
Q&A
How do you manage context when a PR change in one repo affects other systems? Every repo should have a CLAUDE.md or AGENT.md file describing its architecture for the agent to read. ▶ 40:13
Do you block PR merging on AI findings? No. Blocking PRs creates friction. The approach depends on company culture and whether security is seen as an enabler or blocker. ▶ 41:02
Did you face pushback from engineers on AI results? Engineers did not push back on findings. They resisted AI-generated fix PRs due to identity concerns. They wanted the analysis run on all repositories. ▶ 41:52
Should PR security review be separate from general PR review? No. Security should be folded into quality and correctness. One code review process that covers security is better. ▶ 42:57
How do you ensure tech specs are detailed enough to avoid garbage in, garbage out? Context engineering is key. Work with engineering teams to understand where their systems live and how they build software. ▶ 43:45
Notable Quotes
AI is not going to replace security engineers, but security engineers who embrace AI will move faster, solve harder problems, and set the new standard for the field. Anshuman Bhartiya · ▶ 39:20
I think we should think about AI as a force multiplier. Anshuman Bhartiya · ▶ 7:11
The SAS tools don’t do a good enough job. The burden eventually falls on the security engineers to prove what is exploitable and what needs to be fixed. Anshuman Bhartiya · ▶ 8:49
It’s not about coverage holistically. Yes, we will get to the coverage, but I think when you start building a system, the focus should be on what matters more. Anshuman Bhartiya · ▶ 16:52
Key Takeaways
- AI agents found 30 authentication and authorization vulnerabilities missed by all traditional scanners.
- Running the same workflow through two LLMs with consensus voting reduces non-determinism.
- Context engineering with Sourcegraph enables accurate SAST triage across microservices.
- Design review time dropped from days to 5 minutes with AI-assisted spec analysis.
- Start with one high-pain manual process, build a small prototype, and iterate on real findings.
About the Speaker(s)
Anshuman Bhartiya is the Staff Security Engineer and AppSec Tech Lead at Lyft. He has 15-16 years of experience across offensive and defensive security, working at startups and enterprises in cloud security, incident response, and application security. He runs the Boring AppSec community and blogs about his AI experiments.