AI
Threat Modeling as the Original Purple Team Technique
From the threat modeling manifesto: threat modeling is analyzing representations of a system to highlight security and privacy concerns. Four questions drive it: what are we working on, what can go wrong, what will we do about it, and how do we know we did enough. Cox adds a fifth value: it is probably the original purple team technique, because it forces defenders to think like attackers. For AI systems specifically, that attacker mindset is not optional.
Data as an Attack Vector
Around 2012, people called data the new oil. In 2024, data is the attack vector. AI systems run entirely on data, so access to data equals access to the model. A biometric company was breached after hardcoding AWS keys in a public repo; fingerprints and facial metrics were exposed. Unlike passwords, you cannot change a fingerprint. Data provenance matters at scale because any system working on a gradient can be tricked. Whoever touched your training data has had direct access to your model.
MLSecOps: Security Built Into the AI Pipeline
MLSecOps is DevSecOps applied to machine learning pipelines. Cox published what she claims is the first MLSecOps architecture in her paper “Securing AIML Systems in the Age of Information Warfare.” The architecture adds operationalized security reviews on top of continuous monitoring. The core problem: model output can degrade without anyone noticing. Data quality can also degrade through population drift or data poisoning. Without a plan to monitor both inputs and outputs, most serious AI attacks go undetected. Production-grade AI at scale requires this layer.
Three Questions for AI Threat Modeling
Three questions frame AI threat modeling from an engineer’s perspective. First: is it secure? AI is routinely treated as exempt from basic security hygiene. Data scientists rarely ask whether their system is secure or whether it will scale before shipping. Second: can it be operationalized? No plan to monitor data and outputs means security problems exist by default. Survey after survey from major consulting firms shows most organizations still fail to operationalize AI. Third: does it scale? A model achieving 90% accuracy on 7.5 million rows means 750,000 wrong predictions in production.
Mapping the Attack Surface: Data Flows, Provenance, Governance
Three areas to map first. Know your data flows: use data flow diagrams or process flows to show where data lives and how it moves. Know your data provenance: if data came from a broker or from users, outside parties have had direct access to your model. User-generated data is especially dangerous because it effectively hands the keys to your entire system to anyone on the platform. Know your data governance: security professionals are rarely involved, but the regulatory and ethical questions in data governance are security questions.
OWASP AI Exchange as a Structured Starting Point
The OWASP AI Exchange provides controls organized by threat type: threats through use, development-time threats, and runtime application security threats. Cox recommends mapping your data flow diagram against the exchange’s threat categories. Find where data touches users, then look up threats through use. Identify development-time exposure, then match against development-time controls. The exchange brings in experts from multiple fields and is open to everyone. Start with your data flow, cross-reference against the exchange, write policy, and build applications from there.
Q&A
How do you govern Jupyter notebooks to mitigate insider threat? Require check-ins to staged repos and gate model releases on security review; nothing reaches production without that gate, even if fully automating it is operationally difficult. ▶ Watch (36:47)
Are security teams building ML-specific controls, and what’s most overlooked? The most pressing gap is knowing who has access to what; most organizations have ad hoc pipelines where data scientists grab code from arbitrary notebooks and no one tracks where IP lives. ▶ Watch (38:29)
Notable Quotes
AIML at scale is literally impossible Susanna Cox · ▶ Watch (15:44)
this is a security issue Susanna Cox · ▶ Watch (26:54)
please know who has access to this because in many cases Susanna Cox · ▶ Watch (38:50)
Key Takeaways
- Data provenance is a security boundary; whoever controls training data controls the model.
- Without operationalized monitoring, every AI system has security gaps by default.
- 90% accuracy sounds impressive until it means 750,000 wrong predictions at 7.5 million rows.
- Data scientists almost never ask if their model is secure or whether it will scale before shipping.
- The OWASP AI Exchange maps AI threats by phase so security teams can find mitigations by data flow.
About the Speaker(s)
Susanna Cox has spent her career in AI security, applying her background in cybersecurity and aviation to engineering mission-critical AI for aerospace and defense. A former red teamer and data scientist who went on to serve as Chief Data Officer, she holds patents pending in AI security. Her paper “Securing AIML Systems in the Age of Information Warfare” introduced the first published MLSecOps architecture. She is a member of the OWASP AI Exchange core team.