AI
Data Voids and Copilot’s Trust in Bing
Copilot reaches out to Bing’s index to answer questions about topics outside its training data. Tobias Diehl tested this by publishing a fake financial policy called “guilt gift policy” that described money laundering. It took 2 weeks to rank in Bing. Copilot then repeated the content and backed it with legitimate sources. A fake CVE on GitHub ranked in 1 hour. Copilot’s trust in Bing’s ranking makes it vulnerable to data voids.
Key Term Association Attack
Microsoft’s 2018 paper defined data voids as search engine vacuums with no relevant content. Threat actors abused them to spread misinformation about gun violence events. Diehl used a key term association attack. He could not take over the term “Copilot” itself, but he associated malicious content with predictable traffic keywords like installation commands. He used Copilot to generate longtail keyword questions. The payload used Invoke-RestMethod to download a script and Invoke-Expression to execute it. Copilot did not verify the hosted content.
Turning Copilot into a Phishing Proxy
In the first demo, a user asks Copilot about a fake Microsoft product. Copilot fills citations with legitimate Microsoft content and confirms it is secure. When asked how to install, Copilot repeats the initial access command without citations or warnings, requesting admin permissions. In the second demo, Diehl targeted the term “watch zero day quest”. It took 4 weeks to rank. Copilot presented the malicious command as the official way to watch the event. Clicking the citation link led to a Rickroll or a phishing page.
Poisoning a Hacking Competition
During Microsoft’s Zero Day Quest, Diehl poisoned the Copilot flash challenge by publishing a fake CVE that claimed the user ID was already exploited. Bing indexed new content faster than updated content, so he took over the user ID. Researchers saw the poisoned answer, thought the challenge was solved, and quit. Microsoft released an update 2 days later. Diehl also switched his GitHub to private, causing a 404 error. The attack relies on urgency and the inability to verify sources.
Notable Quotes
I can spread misinformation using your AI by controlling the Bing’s index. Tobias “ItsSixtyNein” Diehl · ▶ 02:35
Copilot’s working the way it’s intended to. It’s given you the most relevant information for the question that you asked, but you’re able to spread misinformation on popular topics. Microsoft response (paraphrased by Diehl) · ▶ 02:42
if the top security researchers in the world with Microsoft, you know, the ones that are living and breathing security every day, if they can’t tell the difference between this poison content, what chance do your users stand against these data voids? Tobias “ItsSixtyNein” Diehl · ▶ 19:02
I was able to jump in and take over the user ID itself. Tobias “ItsSixtyNein” Diehl · ▶ 16:00
Key Takeaways
- Copilot’s reliance on Bing’s index allows attackers to poison data voids with fake content.
- A fake CVE on GitHub ranked in one hour; a fake policy in two weeks.
- Microsoft initially deemed the attack out of scope but later added warning messages.
About the Speaker(s)
Tobias Diehl is a security researcher and offensive security engineer with over five years of experience in the financial industry. He is a Microsoft Most Valuable Researcher (MVR) for his contributions to vulnerability discovery and responsible disclosure across Microsoft platforms.