AI
The Old Software Security Model Is Broken
The old model: identify a need, trial a few options, spend months integrating them, pick one, pay the vendor, integrate it everywhere else. Compliance is happy. But the tool is stiff, edge cases pile up, and feature requests take quarters. Security teams end up with a long list of things they wish they had time to do. The vendors know the best practices. So do you. The integration cost is nearly as high as building it yourself.
McMillan put it plainly: we don’t need consultants. We already have the list. We need more engineering hands and less friction.
Code Is Free — Including Security Code
“Code is free. I mean it. Code is free.” — Paul McMillan
Software doesn’t cost anything to build anymore, and that includes security software. If you want a security outcome, add a test to your existing CI/CD pipeline, commit a text file listing what you want checked, and write a tool call that returns zero or one with a short explanation. Feed it back into your CI framework. No vendor contracts, no frameworks. Ryan’s team built a product with a million lines of code, 250,000 of which are prompts, by pulling expertise from subject matter experts and reflecting it directly into the codebase. Any developer already using LLMs to write code can do the same thing tomorrow.
Living Threat Models in the Codebase
Threat models go stale the moment they leave the room. Checked into the repo, validated on every PR, they stay alive. Lopopolo’s rule: if it’s trapped in a Slack thread from three months ago, agents can’t read it and engineers won’t.
“A threat model that’s locked in a Slack thread or some Google Doc that I know I am never going to read is not something that I am able to build durable product with.” — Ryan Lopopolo
Models can deduce a threat model from existing code. Generate one, commit it, then add a required CI step that confirms each PR doesn’t break its assumptions. The whole thing fits in 40 lines of YAML. When a change invalidates a documented risk decision, the agent tags the right human, explains why, and asks the specific question that needs answering.
Encoding Expert Feedback as Executable Guardrails
“Every Slack message I have ever gotten that says there’s a concern I’m not taking into consideration, every PR review comment I have ever gotten, every test failure, every lint error, every dependency I have rolled back, is in some way missing context that has prevented me, the members of my team, and the agents that we have working on our behalf from doing the right thing up front.” — Ryan Lopopolo
Every piece of that feedback is a gap in your guardrails. The fix is not patching the specific issue and moving on. Lopopolo asks: what process failure let this code get written at all? Fix the point issue, then write a cheap bespoke lint that statically disallows that class of mistake. Two sentences in security.md — “secure code comes from secure interfaces that are impossible to misuse” — eliminated entire path traversal patterns. AppSec findings become prompts. PR comments feed a nightly distillation job. The codebase accumulates taste.
Agentic Supply Chain Hardening at Scale
Supply chain hardening stays below the line for most teams. Lopopolo changed that with a 15-minute coffee chat. He spotted a PNPM post-install hardening PR in the Codex repo (14:22), pasted the diff to Codex, and had an equivalent diff for his own codebase 15 minutes later. That same afternoon, he turned a point-in-time audit into a daily scanner forking 16 parallel agents (17:46) to cover all 1,500 direct and transitive dependencies, emit SARIF, and push findings to GitHub Advanced Security. Four hours in a background terminal tab.
Q&A
What is the ratio of tokens spent building versus tokens spent validating and guardrailing? Ryan estimates roughly 50/50 — half the token budget goes to producing code, half to refining it via CI reviewer and scanner agents side-effecting back into the repository through PR comments, GitHub Advanced Security, and GitHub Actions. ▶ 21:31
How are you thinking about insider risk or the ability to bypass these scanners by instructing the model to approve everything? Ryan’s answer: a separate model checks for bad behavior, and a prompt injection that fools the dependency-update agent is unlikely to fool a different model checking whether “the vibes are off” — plus humans still review flagged changes, and two-agent sign-off mirrors the two-party controls used for humans. ▶ 22:58
Have you considered using production insights, telemetry, or observability rather than just the code? Yes — the Codex harness ships a native OTEL exporter used for agent governance, and all session logs are durably archived for DFIR review. ▶ 24:15
Notable Quotes
Code is free. I mean it. Code is free. Paul McMillan · ▶ 1:58
A threat model that’s locked in a Slack thread or some Google Doc that I know I am never going to read is not something that I am able to build durable product with. Ryan Lopopolo · ▶ 4:57
Every Slack message I have ever gotten that says there’s a concern I’m not taking into consideration, every PR review comment I have ever gotten, every test failure, every lint error, every dependency the I have rolled back, is in some way missing context that has prevented me, the members of my team, and the agents that we have working on our behalf from doing the right thing up front. Ryan Lopopolo · ▶ 9:41
Tokens are cheap. Throw an agent at it. Ryan Lopopolo · ▶ 5:38
We deploy 50% of our token budget toward actually producing the code, and then another 50% refining it. Ryan Lopopolo · ▶ 22:10
Key Takeaways
- Check your threat model into the repo and make it a required CI review step on every PR.
- Distill every security finding, Slack concern, and AppSec comment into executable lints, tests, and guardrails — not point fixes.
- Build bespoke, fit-for-purpose security tooling with coding agents in hours, not quarters; the code is free.