AI
Promptware and the Shift from Memory Corruption
Cyber attacks historically targeted memory corruption bugs. Prompt injection works differently. A promptware payload is text, image, or audio deliberately crafted to make an LLM trigger malicious actions against its own user. No GPU cluster required. No white-box model access needed. Expertise level: proficient at most, probably less. The speakers draw a direct analogy to 2015, when attacks against connected cars were called exotic. Charlie Miller and Chris Valasek proved them wrong. This talk attempts the same for LLM-powered systems.
Gemini for Workspace Agent Architecture
Gemini runs on a foundational model (Flash 2.5 at time of research) acting as an orchestrator. It breaks user requests into tasks and executes them in sequence. Short-term memory is the session window. Long-term memory includes emails, calendar events, files, and anything the user asked Gemini to remember. On Android, Gemini is now the default Google Assistant unless changed. The orchestrator can trigger sub-agents for Gmail, Google Calendar, Google Home, Zoom, and Android system utilities including flashlight, calls, screenshots, and website navigation.
Context Poisoning via Calendar Invite
The attack vector is a single Google Calendar invitation with the promptware in the event title or description. When the victim asks Gemini “what are my calendar events,” Gemini fetches all events and loads them into its context window, including events hidden behind the “show more” button. That button is key: Gemini inserts event text into chat history even when the user never clicks it. The malicious text now sits in Gemini’s active context and can override subsequent responses. The first demo showed Gemini recommending a GitHub page and generating toxic content, no guardrails triggered.
Tool Chaining Inside the Calendar Agent
Gemini’s calendar agent can read, create, and delete events. The researchers chained both tools in one shot: after reading events, Gemini immediately deleted a random one without displaying the calendar view to the user first. Each follow-up reply triggered another deletion. A second technique, attention override, exploited how LLMs weight recent context. Gemini almost always opens with “Here are your events for this week.” Injecting that same phrase with different text replaces the reply entirely, bypassing filters because the model treats it as a known learned response being updated.
Delayed Invocation and Physical-World Compromise
Google blocks out-of-scope agents from activating via indirect injection. The bypass: delayed tool invocation. Instead of triggering Google Home immediately, the injection tells Gemini to act only after the user says “thanks.” When that trigger fires, the expected agent becomes Google Home, passing the scope check. Combined with the “show more” persistence trick, the researchers opened apartment windows, activated a boiler, and forced a Zoom session. For Zoom, they wrapped the app-intent URI in an HTTPS redirect via a URL shortener, bypassing Gemini’s URI filter.
Email Exfiltration and Worm Potential
Google blocked one-step URL assembly-and-open. The two-step worked: the injection tells Gemini to construct a URL from target email content and print it to chat history, then open that URL when the user says “thanks.” In the demo, the subject of a second email, “series C round 50 million,” was encoded into the URL and sent as an HTTP GET request to an attacker server. Exfiltrated email addresses feed new invitations, creating a worm that spreads the injection to new Google Workspace users.
Notable Quotes
I call them genius toddlers. Stav Cohen · ▶ 13:30
We are not just hijacking a chatbot. Stav Cohen · ▶ 19:55
It’s a half click. Ben Nassi · ▶ 43:25
Key Takeaways
- A single Google Calendar invite is enough to compromise Gemini’s context and trigger malicious agent actions.
- Promptware attacks need only a target email address and a smartphone, no specialized equipment or adversarial ML expertise.
- 73% of the 14 demonstrated attacks against Gemini for Workspace scored high to critical severity.
- Delayed tool invocation bypasses Gemini’s agent-scope enforcement, enabling physical-world effects via Google Home.
About the Speaker(s)
Ben Nassi is a Black Hat board member (Asia and Europe), a cybersecurity expert, and a consultant. He specializes in AI security, side channel attacks, cyber-physical systems, and threat analysis and risk assessment. His work has been presented at top academic conferences, published in journals, and covered by international media. Ben is a frequent speaker at Black Hat (6 times), RSAC (2 times), and DEF CON (3 times), and won the 2023 Pwnie Award for Best Crypto Attack for Video-based Cryptanalysis.
Or Yair (@oryair1999) is a security research professional with seven years of experience, currently serving as Security Research Team Lead at SafeBreach. His primary focus is vulnerabilities in Windows operating system components, with past work covering Linux kernel and Android components. Or’s research is driven by a commitment to challenging conventional thinking and considers creativity a core skill for the field. He presents regularly at Black Hat, DEF CON, RSAC, SecTor, and other top conferences.
Stav Cohen is a Ph.D. student at the Technion – Israel Institute of Technology, investigating Cyber-Physical Systems that integrate GenAI methodologies and feature Human-in-the-loop interactions, with a focus on security and operational aspects. He conducts detailed analyses of GenAI models to identify potential vulnerabilities and devise mitigation strategies, and explores how GenAI can improve both security and operational efficiency in Cyber-Physical Systems.