AI
HuggingFace: From Chatbot to AI Infrastructure Hub
In 2016, three founders raised $1 million for an AI chatbot. Google’s Transformers paper dropped in 2017, then BERT in 2018. HuggingFace adopted BERT, released PyTorch pre-trained BERT 0.1.1, and later launched the hub. The Stack dataset alone is 3 terabytes of permissive code from GitHub. Printed, it would fill 288 miles of shelf space. The Stack is one of 883,000 datasets on HuggingFace, and it is not the largest.
The Nanix Data Leak: 1,800 Private Repos Exposed
A Nanix employee created a dataset of Semgrep scans and GitHub issues to train AI on vulnerability detection. They used a classic personal access token with repository scope. The token gave access to all 1,800 private Nanix repositories. The scrape included those private repos. Semgrep saves secrets in cleartext. The entire 10 GB was Apache 2 licensed and downloaded thousands of times. The dataset was taken down, but copies persist.
ComfyUI: API Keys Embedded in Image Metadata
ComfyUI, a stable diffusion workflow tool, embeds the full prompt and workflow JSON into PNG metadata by default. Users who called third-party APIs had their API keys baked into every image they distributed. TruffleHog flagged live keys inside PNGs on HuggingFace. The metadata section contained the keys in plaintext.
A JetBrains Intern’s Leaked Token
A 2024 JetBrains intern created 20 GitHub accounts with 20 API keys to scrape patch data for a bug-fixing AI dataset. They used one real account token as a shortcut. When they zipped the cloned repositories, the .git/config file containing that token was included. The dataset was published and copied thousands of times. The real token had access to private repositories.
Stolen Stripe Keys in Public Telegram Channels
A data scientist named Tom scraped 10 million messages from public Telegram channels. Buried inside were channels run by a criminal enterprise that rewarded people for submitting stolen Stripe keys. The keys were used to test stolen credit cards. The reward system included signup bonuses and giveaways. Ayrey’s team got those Stripe keys revoked a week before the talk.
Why Open Data Is Impossible: Copyright, Privacy, Security
The Open Source Initiative suspended its board election over whether data must be open for a model to be called open source. Common Crawl, a 363-terabyte scrape of the entire internet, is evaporating as copyright holders send takedown requests. Models trained on it cannot be reproduced. Leon5B, a dataset of 5 billion image links, avoids copyright but links rot. Ayrey found credit card numbers, social security numbers, and copyright violations across the hub. The data is too large and too replicated to ever clean up.
Notable Quotes
“one exposed API key is a tragedy. Well, hundreds of thousands just becomes a statistic.” Dylan Ayrey · ▶ 7:47
“the data sets that are on HuggingFace comprise almost all of humanity.” Dylan Ayrey · ▶ 33:34
“there’s no going back.” Dylan Ayrey · ▶ 34:26
Key Takeaways
- HuggingFace hosts 883,000 datasets; scanning 25% found 100,000 live secrets.
- Private data leaks into public datasets through misconfigured API tokens and default metadata embedding.
- Data curation biases push more live API keys into training sets, creating a feedback loop.
- Copyright, privacy, and security issues make opening AI training data legally and practically impossible.
- The OSI suspended its election over the question of whether data must be open for open-source AI.
About the Speaker(s)
Dylan Ayrey is the original author of the open source version of TruffleHog, which he built after recognizing just how commonly credentials and other secrets were exposed in Git. Coming most recently from the Netflix security team, Dylan has spoken at a number of popular information security conferences.