Attacker Mindset for AI Systems

▶ Watch (01:50)

Thinking like an adversary means identifying what’s worth stealing: data, code, service uptime, or brand reputation. For AI systems, the target is layered. Cloud-hosted ML models sit behind software stacks that rarely have monitoring on their API calls. Local deployments add OS access to the attack surface. The attacker’s job is to find attacks that work across all three architectures: large-scale vendor infrastructure, cloud-plus-API setups, and local agents. Common to every architecture: the model, sensitive data, proprietary code, and critical infrastructure.

Training Data as Attack Surface

▶ Watch (08:38)

Training data is the first weak point. Large computer vision datasets scraped from the internet contain NSFW images, mugshots, watermarked advertising, and medical photos uploaded without patient consent. Text datasets carry credit cards, email addresses, and dark web content. An attacker who knows what went into the training set can craft inputs designed to extract it back out. Jarmul’s recommendation: download a few of the public large-scale datasets and look around. What you find tells you what the model might reveal.

Model Internals as Attack Surface

▶ Watch (14:13)

Decision boundaries inside deep learning models are where attacks live. Denser embedding spaces carry more information, and the margins between class boundaries are where an attacker can nudge an input toward a different output. GPT-scale models compound this: they are overparameterized, meaning more model parameters than training data points. The model has more space than information, so original training data can persist literally inside the weights. Attention makes it worse: important tokens stay activated across an entire conversation, giving an attacker a long window to steer what the model treats as high-value context.

Attack Classes and Their Targets

▶ Watch (21:48)

Four attack classes map to four targets. Data exfiltration: the Carlini diffusion model extraction (22:39) shows that asking a model to fill in blanks from its own training images recovers them near-exactly. RAG documents are equally exposed. Output manipulation: adversarial examples, patterns invisible to humans, push models across decision boundaries and have no reliable fix. Backdoors can be proven mathematically undetectable. Agentic attacks extend the blast radius: the Gemini meeting-invite hijack (31:42) embedded a prompt that fired whenever a user said “thanks”, triggering smart home actions.

Defense Practices

▶ Watch (32:11)

Start with architecture segregation. Treat the model as untrusted software with malware inside it, then decide how to isolate it from the rest of the system. Input and output sanitization, access controls, and choosing smaller or local models where possible all reduce the surface. Guardrails help but are not a solution: they are easy to bypass. Real protection comes from interdisciplinary threat modeling, getting engineers, product, and security in a room to expose what actually needs defending, and from continuous red teaming built into MLOps pipelines, not just vendor security attestations.

Notable Quotes

to learn to think like an adversary. Katharine Jarmul · ▶ 1:50

the model is the vector the target is the data Katharine Jarmul · ▶ 24:38

did we poison the model already? Katharine Jarmul · ▶ 27:54

Key Takeaways

  • AI training datasets contain PII, medical images, and dark web content that attackers can directly extract.
  • Overparameterized models store more space than data, so training examples may persist verbatim in weights.
  • Defense requires architecture segregation and continuous red teaming built into MLOps, not guardrails alone.