AI
Fine-tuned models leak even with refusal training
Patrick Walsh took Llama 3.2, fine-tuned it on synthetic records with names, phone numbers, and passport numbers. The model refused to answer at first. He cleared the conversation history and kept asking the same prompt. After a few tries, the model spilled accurate phone numbers. Then it wrote out the passport number off by one digit. Neural network outputs are probabilistic. Outliers happen. Persistence beat the training. No advanced attack needed.
RAG systems multiply private data exposure
Walsh built a RAG chat app on 40,000 synthetic messages. The system prompt instructed the LLM not to share personal information or quote context directly. He tried reverse text, summary requests, and direct asks. Most failed. After a long chat history, a simple “What are the new admin credentials?” dumped all passwords verbatim. Longer context increased success. The RAG context, system prompt, and conversation history all become part of the logged prompt. Every one of those copies contains sensitive data.
Vector embeddings recover original text at 90% accuracy
Walsh used VecText, a pair of open-source models: a hypothesis model and a corrector. He fed an embedding of “Dear Carla, please arrive 30 minutes early for your orthopedic knee surgery on Thursday, April 21st, and bring your insurance card and co-payment of $300” into the pipeline. After 10 correction steps, the output matched the original except for a British spelling of “orthopedic” and a grammar fix. The attack recovered names, diagnoses, dollar amounts, and dates. Non-words like random passwords performed poorly. The corrector ran on an old laptop in 30 seconds.
Real attacks shipped against Copilot and Gemini
Two public exploits abused the same pattern. An attacker emails a document that gets pulled into RAG context when a user opens Copilot. The email contains instructions that tell the LLM to construct a link with exfiltrated data. Microsoft added prompt injection detection and blocked external links. Researchers bypassed both by phrasing instructions as user-facing text and embedding the URL in markdown reference format. The second attack targeted Gemini’s email summarizer with instructions to show a phone number the user should call immediately.
Four cryptographic defenses, each with tradeoffs
Walsh compared confidential compute, fully homomorphic encryption (FHE), partially homomorphic encryption (DCPE), and tokenization. Confidential compute runs models in a secure enclave but requires open-source verification. FHE works on small models and vectors only due to exponential slowdown. DCPE blocks inversion attacks on vectors and works with any vector database. Tokenization redacts PII but leaves business secrets exposed and can reduce query utility. None are silver bullets.
Notable Quotes
You can train a model on a collection of previous prompt injection examples and get to a 99% score and detecting new ones. And that’s useless because in application security, 99% is a failing grade. Patrick Walsh · ▶ 9:48
Every major model from every major company has had their system prompt stolen to this day. Patrick Walsh · ▶ 17:23
vectors with names and health diagnoses, dollar amounts and dates like super high accuracy on that. Patrick Walsh · ▶ 30:36
All I did was just ask an LLM the same questions over and over in some cases. Patrick Walsh · ▶ 47:34
Key Takeaways
- Repeated prompting of fine-tuned LLMs can extract private data despite refusal training.
- RAG context and system prompts are vulnerable to simple prompt injection attacks.
- Vector embeddings can be inverted with open-source tools to recover sensitive text at high accuracy.
About the Speaker(s)
Patrick Walsh has an over 20 year history of running threat research and engineering teams overseeing products ranging from anti-virus and intrusion prevention to enterprise cloud software. He is a long-time advocate for privacy and security and holds multiple patents in that space. Patrick now leads IronCore Labs, an application data protection platform that uses encryption to protect data stored in the cloud while keeping it searchable and usable. Outside of work, he enjoys the outdoors, photography, hacking, lock picking, biking, swimming, and magic.