AI
Prompt as Code — Why GenAI Breaks Traditional Security Boundaries
Traditional app security filters on syntax — a <script> tag, a SQL injection pattern. GenAI breaks that model entirely. Instructions are semantic, not syntactic. They can be obfuscated in dozens of ways that standard filters miss. LLMs are also context learners: they treat system instructions, user input, and external data as one contiguous token stream. There is no NX bit for the context window.
“the prompt is code like every single token in the input stream is a potential instruction” — Nicolas Lidzborski
That collapse of the code/data boundary is structural. Injected data can overwrite instructions because the model has no out-of-band way to distinguish the two.
Indirect Prompt Injection and the Three Primary Attack Vectors
Indirect prompt injection is a zero-click attack. The malicious instruction hides in a calendar invite or email. When an agent summarizes the inbox, it ingests that hidden text and executes the attacker’s command. The Gmail hidden-payload demo (05:22) shows a payload invisible to the user overriding the agent mid-task. The LLM treats system instructions and external data as one token stream, so untrusted content overwrites the agent’s directives.
The Google Calendar invite attack (09:55), from Black Hat, shows the consequence: a hidden invite payload triggered lateral movement and home-automation control the moment a user asked “what’s on my schedule?” No malware needed.
Why Reactive Filtering — Including LLM-as-Judge — Always Loses
Static filters fail first. Block lists and regex match on syntax, so ROT13 or hex encoding breaks the pattern and the payload passes through. ML classifiers do better, but synonym swapping or translating to a low-resource language shifts confidence just enough. Text-only filters miss instructions buried in image metadata entirely. Each layer has one bypass. Attackers only need one.
LLM-as-judge adds a second model to review responses before they reach the user. The flaw is structural: the judge reads the same untrusted data as the primary model, through the same semantic interface. A payload can target the judge directly (13:15) — instructing it to rate malicious content as safe “for research.” The review layer becomes the attack surface.
“you do not want to play cat on mouse because you’re always behind and you’ll get exploited” — Nicolas Lidzborski
A Layered Structural Defense Blueprint
Google’s defense runs four layers. First: strip untrusted inputs before the LLM sees them — visible email text only, no hidden HTML, known phishing blocked outright. Second: mark data boundaries inside the prompt with sentinel tokens and fine-tune the model to ignore imperative commands within those delimiters. Third: move policy enforcement out of the LLM into a deterministic orchestration layer. A finite state machine tracks context risk at every step and can block, for example, any web request that follows access to sensitive user data. Fourth: scrub the output. Markdown renders as HTML, so anomalous image links and hallucinated URLs get stripped before reaching the browser.
Systemic Resilience — Testing, Human-in-the-Loop, and Feedback Loops
Static defenses break. Lidzborski’s answer is a continuous testing loop: automated regression tests replay every known agentic attack class after each feature ships, and a bug bounty program hunts zero-day vectors before attackers do. Because LLMs are probabilistic, the same attack can pass on attempt ten even when the first nine were blocked — so the framework repeats each attack many times. For high-stakes actions, a plan-validate-execute gate requires human confirmation before anything runs.
“we cannot secure genai as a oneshot with one time measure it’s a continuous and selfcorrecting process” — Nicolas Lidzborski
User signals close the loop. When users report a suspected injection, those reports tune the detection service — the same model Google applied to Gmail spam.
Q&A
LLMs repeat the von Neumann mistake of mixing code and data in one memory space — why not use two separate prompt windows, one for commands and one for data? Lidzborski agreed the structural parallel is real but didn’t engage with the two-window proposal directly, pivoting instead to the broader problem of a fast-moving industry building insecure systems by default — citing a 30-day median enterprise patch time for critical vulnerabilities as evidence the gap is already dangerous. ▶ 22:51
Notable Quotes
the prompt is code like every single token in the input stream is a potential instruction Nicolas Lidzborski · ▶ 2:16
you do not want to play cat on mouse because you’re always behind and you’ll get exploited Nicolas Lidzborski · ▶ 13:58
we cannot secure genai as a oneshot with one time measure it’s a continuous and selfcorrecting process Nicolas Lidzborski · ▶ 22:03
it’s very interesting how much we are going back in time and we are back in the time of Windows 95 Nicolas Lidzborski · ▶ 23:52
Key Takeaways
- Replace reactive prompt filters with structural, layered defenses — filtering is permanently behind the attacker.
- Track data provenance throughout every agentic workflow and enforce deterministic policy at each orchestration step.
- Treat GenAI security as a continuous loop: automated regression tests, user feedback signals, and human confirmation gates.