AI
AI Agent Autonomy Levels and Attack Surface
Not every LLM app carries the same risk. Rich laid out three levels. Level one: a fixed pipeline where one input always triggers the same control flow. The LLM cannot change what runs next. Level two: a router that picks among a bounded set of workflows. Level three, fully autonomous, is what most people call an agent. Claude, ChatGPT, and Cursor all run this way. The agent plans, picks tools, and acts without user input. Put something in and you have no idea what comes out.
That non-determinism is the attack surface. Higher autonomy means more paths an attacker controls.
The Universal Anti-Pattern Behind Every AI Attack
Every AI attack on an LLM-backed agent follows three steps. Untrusted data enters the agent. An LLM parses it and produces attacker-controlled output. That output reaches downstream tools with real capabilities. Lynch called this the universal anti-pattern, and it covers everything from RAG poisoning to direct prompt injection.
The attacker’s job is to answer three questions: Where can I insert my data? How does the model parse it? What tools sit downstream? Delivery format matters. Email, markdown, code comments, images — the right channel depends on how the pipeline ingests content. But none of that complexity changes the core condition:
“as long as step one is true and an attacker is able to get their untrusted data in front of your agent in some way, you should assume that they can potentially control anything downstream from that model.” — Rebecca Lynch
Prompt Injection via RAG and Code Execution Agents
Microsoft Copilot reads user email and SharePoint. One malicious email puts attacker instructions into the RAG database. The Microsoft Copilot email exfiltration via markdown image (08:52): white-on-white text tells Copilot to collect credentials, then renders an attacker-hosted image that ships them as URL parameters. Microsoft patched URL rendering after disclosure.
PandasAI generates and runs Python on the host. Its system prompt is public, so the team spoofed a guardrail check and fed pre-written code to the executor. The PandasAI guardrail bypass and reverse shell (11:35) delivers a base64 payload via a native Pandas call — reverse shell on the server.
Computer Use and Vibe-Coding IDE Exploits
Computer use agents loop: screenshot, server reads it, bash commands execute. Every page they visit is an injection vector. Lynch planted a fake GitHub issue with obfuscated PowerShell. One developer prompt was enough: the agent read the issue and popped a shell (16:36).
Cursor carries the same risk. A Unicode tag-space payload hides a shell command inside a crowdsourced .cursor rules file (20:42); the agent social-engineers the user into approving it. With auto-run on, a malicious PR’s setup.py spawns a reverse shell while the agent reviews dependencies (23:21). Cursor’s response: no CVE, this is how agents that execute code work.
Securing AI Agents: Defense Principles and Mitigations
Treat prompt injection as the baseline, not an edge case. Map the kill chain: where does untrusted data enter, where does the LLM get hijacked, and what tools fire with attacker-controlled output. A poisoned level-three agent has access to every tool a user does — effectively an insider threat. The three mitigations that cut the most risk: taint-track untrusted data away from sensitive tool calls, scope tools to the least needed, and run command execution in isolated containers. Cursor’s background agents demonstrate the container model working in production.
“least autonomy is the new least privilege. And when you are designing agents, assume prompt injection is the new assume breach.” — Rich
Notable Quotes
the LLM that drives and controls your agents can potentially be controlled by attackers. Rebecca Lynch · ▶ 1:26
as long as step one is true and an attacker is able to get their untrusted data in front of your agent in some way, uh you should assume that they can potentially control anything downstream from that model. Rebecca Lynch · ▶ 8:34
least autonomy is the new least privilege. And when you are designing agents, assume prompt injection is the new assume breach. Rich · ▶ 39:03
these autonomous agents, if they have their goals replaced by an attacker, they will work just as hard for the attacker as they will for you. Rich · ▶ 34:25
Key Takeaways
- Treat every external data source your agent reads as an active prompt injection vector, not just user input.
- Restrict agent autonomy to the lowest level that accomplishes the task; full autonomy hands attackers a ready-made insider threat.
- Isolate command execution in containers and taint-track sensitive tool calls from any context that has touched untrusted data.