AI
Why Prompt Injection Cannot Be Solved
Prompt injection has no separation between instructions and data. SQL injection was fixed in 1998 with prepared statements because databases distinguish code from data. LLMs treat everything as tokens. The speaker called it an unsolvable problem like social engineering a human. Five Fortune 500 companies had this same vulnerability. Guardrails reduce but cannot eliminate the risk.
Embedding AI Creates a Dual Supply Chain Risk
AI agents in CI/CD pipelines automatically process untrusted input from pull requests and issues. This adds a new layer to the traditional supply chain of developer code, dependencies, and build systems. The orchestration layer gives AI the ability to make decisions on untrusted data. Attackers can inject malicious instructions that the AI executes using its tools and permissions.
The Google Gemini Vulnerability: Prompt Injection to Leak Tokens
Google’s Gemini repository used a GitHub Action to triage issues. The YAML file passed the issue title and body to Gemini CLI. The AI had environment variables with the Gemini API key and GitHub token. Core tools like “edit issue” let the LLM modify issues. A prompt injection told the AI to read those tokens and publish them in the issue. The demo showed the tokens leaked.
How Google Fixed It: Remove Tokens and Core Tools
The fix does not prevent prompt injection. Google removed the LLM’s access to secrets by running Gemini CLI without authorization tokens. They also stripped core tools, leaving only the echo command. The blast radius shrinks even if injection succeeds. The speaker emphasized that you cannot stop the model from being injected; you limit what it can leak.
Threat Model AI Agents as Confusable Humans
Treat AI agents as human proxies that can be socially engineered. Three principles: restrict access to secrets and tools, validate inputs and outputs with guardrails, and reduce the blast radius by limiting permissions. Threat model the full workflow: trigger, input, model, tools, secrets. These steps allow safe use of AI in CI/CD without assuming the model can be trusted.
Notable Quotes
the AI agents are the new control plane in our in our model and it becomes a decision maker Mackenzie Jackson · ▶ Watch (24:23)
embedded a little human into our CI/CD pipeline and that human can be confused Mackenzie Jackson · ▶ Watch (25:10)
We just need to to have an embedded AI system in there, and then we have malicious instructions Mackenzie Jackson · ▶ Watch (30:21)
prompt injection is kind of unsolvable with the current AI architecture Mackenzie Jackson · ▶ Watch (32:14)
Key Takeaways
- Prompt injection in CI/CD leaks secrets because LLMs cannot separate instructions from data.
- Google fixed the Gemini vulnerability by denying LLM access to tokens and removing edit tools.
- Threat model AI agents as confused humans; apply least privilege and validate all inputs and outputs.
About the Speaker(s)
Mackenzie Jackson is a developer advocate with a passion for DevOps and code security. As the co-founder and former CTO of a health tech startup, he learnt first-hand how critical it is to build secure applications with robust developer operations. Today as the Developer Advocate at GitGuardian, Mackenzie is able to share his passion for code security with developers and works closely with research teams to show how malicious actors discover and exploit vulnerabilities in code.