AI
Why AI-Assisted Data Collection Requires Deliberate Prompting
Asking Claude to “grab all these repositories by name” produces working code. It also produces code with no caching, no idempotency, no rate-limit handling, no parallelization, and no backoffs. The script runs. The requirements are missing. A naive prompt gets you naive output.
“you have to be thoughtful” — Rami McCarthy
The fix is an RPI loop: ask AI to research, then plan, then implement. Claude’s plan mode encodes this in a few lines of prompt that stop the model before it writes a single line of code. A repository called superpowers formalizes a seven-step version of the same idea. The specific tool matters less than the habit: structure the prompt before you generate the code, and save composable utilities as you build them so the investment compounds.
Using AI for Signal Extraction and Data Shape Analysis
After scraping, McCarthy had 250,000 flat files and 30 GB of leaked data. Linear scanning hit I/O limits. The first AI win was fingerprinting: 30,000 repositories, but CI/CD pipelines repeat the same job. One Claude-generated fingerprint collapsed that to 13,000 unique machines, 77% CI/CD runners.
“flat files are the final abstraction of AI, I think is is untrue” — Rami McCarthy
AI will not question the abstraction you hand it. Give it flat files and it works with flat files, never suggesting Parquet or an index. You decide the data shape first, then ask AI to extract signals: environment variables, container indicators, package ecosystems, CI/CD platform fingerprints.
Victim Attribution with Reasoning Models — Creativity and Credulity
McCarthy fed all 13,000 repository names into Gemini as one merged string and got back the 10 most prominent companies. Reasoning models know Fortune 100 membership, government domains, and recent VC fundraises. One model spotted that an encoded JWT contained identity claims worth extracting, something McCarthy said he wouldn’t have thought to check.
“the double-edged sword here creativity and credul” — Rami McCarthy
The credulity side failed badly. The string “nucleus” got pinned to a specific company, and anyone running Azure DevOps was tagged as a Microsoft victim. Fix: inject skepticism explicitly, or run a skeptical agent alongside attribution.
Feedback Loops — Distilling AI Samples into Deterministic Rules
Running AI against 30 GB iteratively is slow and inconsistent. McCarthy’s fix: sample with AI, extract a signal, encode it into a deterministic script, then re-run. One pass found a variable holding self-hosted Git instances. That single rule surfaced a Russian e-commerce company, a Thai fintech, and a US Fortune 500. Two weeks of manual work found 200 victim companies. Two days of this cycle found 2,400. The Live attribution skill demo — identifying Alibaba from leaked secrets (19:26) shows the model connecting a dot-file to an internal Alibaba AI developer tool unprompted. The Security contact skill — automated VDP/bug-bounty lookup (20:23) then checks bug bounty programs and VDPs in parallel, replacing 30 minutes of manual searching.
Q&A
How do you handle the tension between ephemeral throwaway tools and long-term reusability? McCarthy said throwing away code is fine for one-off tasks since it can be regenerated, but tools built for a team’s weekly process deserve more thought — and the real shift is realizing you can now put a clean UX on any process you’re about to do manually. ▶ 21:43
What is the token cost for running this kind of agentic attribution pipeline? He estimated roughly $80 for the full run, kept low by using deterministic methods to avoid feeding 30 GB into the context window each time, though cost would scale linearly with more samples. ▶ 22:46
Notable Quotes
you have to be thoughtful Rami McCarthy · ▶ 2:59
flat files are the final abstraction of AI, I think is is untrue Rami McCarthy · ▶ 5:35
the double-edged sword here creativity and credul Rami McCarthy · ▶ 10:57
37 of the Fortune 100 had some level of impact from this incident Rami McCarthy · ▶ 17:46
Key Takeaways
- Use an RPI loop — research, plan, then implement — before asking AI to generate any code.
- Distill AI-discovered patterns into deterministic scripts so findings are reproducible and back-testable at scale.
- Inject explicit skepticism into attribution workflows; LLM credulity produces dangerous false positives in security contexts.