AI
Goose AI Agent as a Red Team Target
Goose started as an internal project at Block in fall 2024 and has since been open-sourced. It ships with a developer shell tool call by default, meaning it can run arbitrary shell commands on the host machine. It connects to a model of the user’s choice (Claude 3.5 Sonnet and 3.7 were common at the time) and integrates with Block’s internal MCP marketplace for tasks like updating Jira tickets or reading Google Drive. Block’s red team saw a clear objective: prove end-to-end compromise of an employee laptop, starting from outside the company.
“I want to emphasize the word operationalize because yes, it’s possible to make a prompt injection. We all know that. But can you make one without getting caught, without getting detected? Can you make one that hides in plain sight?” — Josiah Peedikayil
Campaign One: Indirect Prompt Injection via Google Calendar MCP
Block’s internal MCP marketplace accepted calendar invites from external accounts, letting an attacker plant untrusted text into Goose’s context. The team built a curl-bash payload, hid it in an invite description, and suppressed the email notification so the invite appeared silently with the attacker’s account omitted. The visible-payload problem they solved with Invisible Unicode payload in Google Calendar invite (08:14): zero-width Unicode characters buried the full injection, invisible to humans but processed by Goose. When a user asked “Hey Goose, what’s on my calendar today?” Goose would read the description, hit the injection, and chain a developer shell tool call to execute it. Invites went out. No shells came back.
Operational Failures: Model Variance, Context Dilution, and Testing Gaps
Campaign 1 failed on four fronts. First, the Google Calendar MCP was rewritten mid-operation, breaking a payload tuned to the old version. Second, the injection worked on Claude 3.5 Sonnet but fell apart on Claude 3.7 and GPT-4. Third, real employee calendars hold 20-plus invites — the model ingested all of them, context filled up fast, and the one invite with the shell command got dropped. Fourth, the team had no automated testing framework to measure injection success across models before launch, so the failure modes were invisible until the campaign was already live.
Campaign Two: System Prompt Injection via Goose Recipes
Goose recipes load directly into the system prompt, a higher-privilege target than a calendar description. The team embedded a curl-bash payload in the instructions field and hid it with zero-width Unicode. Goose recipe system prompt injection demo (12:59): Goose opened Chrome to a Rick roll.
With Calendar sends capped at 50 per day, they added spear phishing. Posing as external researchers, they filed a fabricated RTL text bug report with the Goose dev team and asked a developer to open the recipe. He did. Spear phishing the Goose dev team via fabricated RTL bug report (16:45) triggered the curl-bash payload.
Mitigations: Hardening AI Agents Against Prompt Injection Kill Chains
Block stripped non-standard Unicode characters from all inputs and recipes first. Then they added a recipe warning screen that shows users exactly what a recipe will do before they run it — previously the Base64 encoding meant you had no idea without decoding it yourself. For command-level detection, they built a BERT classifier to flag suspicious bash commands and layered an LLM-based semantic check on top of that, though results have been mixed. Command allow-listing cut the attack surface further. For Google Calendar specifically, external senders can no longer book meetings directly; the employee must approve an email first before a calendar invite lands.
Q&A
Did you test the user interface for vulnerabilities like HTML injection, markdown rendering, or other insecure output issues unrelated to tool call control? They did find those cases early in the operation but shared no details, noting that insecure output issues were common across AI agents at launch. ▶ 20:11
Notable Quotes
I want to emphasize the word operationalize because yes, it’s possible to make a prompt injection. We all know that. But can you make one without getting caught, without getting detected? Can you make one that hides in plain sight? Josiah Peedikayil · ▶ 4:22
recipes stood out to us because they go in the system prompt and that really stood out to us Josiah Peedikayil · ▶ 12:33
we had a typo and it was really unfortunate cuz in testing the typo was being fixed by the LLM. It was fixing it for us so in testing it worked but when we deployed it actually it didn’t work Josiah Peedikayil · ▶ 15:49
Key Takeaways
- Operationalizing a prompt injection means hiding in plain sight — evasion discipline, not just technical correctness, determines whether your campaign succeeds.
- System-prompt-level injection via Goose recipes consistently outperforms user-prompt injection; target the highest-privilege context available.
- Automated payload efficacy testing across model versions is not optional — manual tuning for one model silently breaks against another.