AI
The Monorepo as Single Source of Risk
Shopify runs hundreds of services on Kubernetes across close to 500 clusters worldwide. Developers push nearly 60,000 deploys each week and merge more than 1,000 PRs per week into one infrastructure monorepo. That monorepo holds every Kubernetes manifest, Helm chart, deploy configuration, and policy. One bad line in a shared Helm chart gets rendered into hundreds of manifest files by the templating step. A single misconfiguration can ripple from one chart through every environment it touches, all the way to production.
How Misconfigurations Reach Production
Privileged containers set to true grant full host access. Missing CPU and memory limits let a single pod starve every other workload on a node. These patterns enter the codebase through copy-paste or oversight, not malice. Because Helm multiplies one chart into hundreds of manifests, a two-line mistake scales automatically. At over 1,000 PRs per week, manual review could not keep pace. Misconfigurations were reaching staging and production and causing incidents before the team built automated controls.
Semgrep in CI: Three Rules That Block the Worst Cases
Shopify added Semgrep, an open source static analysis tool, directly into CI. Three rules cover the highest-severity cases: no privileged containers, required CPU and memory limits on every container, and no host network, host PID, or host IPC access. Each rule runs on every PR before merge. When a violation appears, the developer receives a clear inline message and self-serve documentation explaining why the setting is dangerous and what exact change fixes it. Blocking is reserved for the most critical patterns; others surface as warnings.
AI Cuts 38,000 Findings Down to Actionable Signals
Early Semgrep rules used overly broad patterns and produced close to 38,000 findings. That volume is not actionable and risks crashing the CI pipeline. The team used AI to review findings, question whether each matched real risk, and refine patterns to be more granular. After multiple iterations, findings dropped by 88% to true positives. The current workflow prompts AI to draft a rule, a human reviews it, AI runs it locally against the monorepo, and the rule ships only after the violation count is acceptable. A dry-run pipeline then reports hits to a Grafana dashboard before any developer is notified.
Results and Lessons After Six Weeks
The team shipped more than 50 Semgrep rules and 13 OPA policies in 6 weeks. OPA handles Terraform plan evaluation where Semgrep scans templates. Together, the controls caught more than 1,000 misconfigurations that developers fixed before merge. Zero incidents occurred after rollout, and no rule required reverting. Four lessons shaped the approach: always start in warn or dry-run mode, drive false positives to a manageable volume, give developers exact remediation steps, and make the secure path the easiest path. When that last condition holds, developers follow it without being blocked.
Notable Quotes
A single change here can ripple across everything. Jie Wu · ▶ 02:44
Basically, the walls of the container disappear. Jie Wu · ▶ 07:45
we were able to reduce the findings by 88% to actionable true positives, and that represents the actual risk. Pulkit Garg · ▶ 17:03
when the secure path is the easiest path, developers will follow it. Jie Wu · ▶ 20:58
Key Takeaways
- One misconfiguration in a shared Helm chart multiplies into hundreds of deployed manifests.
- Semgrep in CI caught over 1,000 misconfigs before merge with zero post-rollout incidents.
- AI refinement reduced 38,000 noisy Semgrep findings by 88% to real, actionable risks.
- Semgrep handles template and manifest scanning; OPA evaluates Terraform plan outputs.
- Self-serve documentation with exact fix instructions keeps developers moving without being blocked.
About the Speaker(s)
Jie Wu is a Senior Security Engineer on the Infrastructure Security team at Shopify, focusing on security automation, IAM, threat detection, and compliance. Before Shopify, she worked on cyber defense initiatives and vulnerability management at Bank of America.
Pulkit Garg is a Security Engineer on the Infrastructure Security team at Shopify, specializing in cloud security, compliance, and supply chain security. Before Shopify, he worked on 5G network software at a startup.