AI
Which Model Does What in Security Work
Haddix uses Claude for code and GPT-4o for writing. For bot development, Anthropic’s models win: they hallucinate less, and project spaces simplify context management. Open-source Llama variants handle internal deployments where data can’t leave the network, but they lag behind cloud models in capability. For companies already inside Azure, Haddix’s argument is direct: Microsoft already handles enterprise data under ironclad contracts, so running GPT on Azure OpenAI service adds little new risk.
Building Security Bots: A Structured Prompt Framework
Haddix’s bot-building method has five core steps: give the bot a persona with expert identity, add “weird machine tricks” (prompts like “take a deep breath” that each push output quality 1-2%), break the problem into step-by-step instructions, list related research terms that tune the model toward expert-level answers, and define output format. He runs the whole design through “system bot,” which writes the system prompt on demand. The result beats single-shot prompting by 30 to 35%.
Attack Surface Discovery Through AI Inference
Two recon bots, both shipped to the ChatGPT store. Subdomain Doctor takes a list of known subdomains (tested during authorized T-Mobile bug bounties) and infers predictable naming patterns: numbered sequences, thematic names, contextual variants. The acquisition bot, pointed at Tesla, queried Crunchbase, Wikipedia, and GPT-4o’s training data. It surfaced smaller acquisitions that Crunchbase did not list, because GPT-4o’s training set captured deals that never made news. Both bots can run from the command line via automated scripts.
OSINT-Driven Spearphishing Generation
A framework demonstrated at Defcon’s Social Engineering Village deployed a team of AI agents across social networks, people-search sites, and data brokers to profile a target, then generated a personalized phishing email. Tested live against Dave Kennedy, former NSA and head of a major red team consultancy, the system produced an email from Tiffany Broch, the real bodybuilding.com Community Development project manager, offering Kennedy a leadership role in a new Cleveland chapter. Kennedy confirmed he would have clicked it.
SOC Automation: Query Writing to Playbook Design
Blue team use cases start at the SOC manager layer. A briefer bot writes incident-update emails for executives from a short description. A tabletop designer generates custom IR exercises without the budget of a Big Four engagement. An IR playbook bot creates role-specific checklists for active incidents. On the analyst level, an ELK bot writes Elasticsearch queries, a Splunk bot generates Splunk SPL, and an osquery bot hunted for DLL side-loading during a live engagement. A Yara bot trained on the 100 Days of Yara project creates malware detection rules.
Leadership Bots and Purple Team Integration
For security leaders, Haddix built a boardroom bot drawn from his time as CISO at Ubisoft. It argues against his security budget proposals, forcing him to sharpen justifications before bringing them to executives. A tech-explainer bot translates security jargon for non-technical audiences. A next-gen CISO bot encodes his security leadership philosophy and stress-tests program decisions. On the purple team side, MITRE bots and a Semgrep bot fed with Trail of Bits’ advanced CodeQL guide complete the curriculum. The two-day class runs quarterly.
Q&A
Have you heard of WormGPT or similar unfiltered models for security work? Haddix prefers context-based prompt tricks (like framing requests as CTF problems) over unfiltered open-source models, which lose quality when post-training alters their base embeddings. ▶ Watch (1:01:44)
How do I access the full two-day training? The class runs quarterly, with the next date expected in early December, posted on Arcanum’s website. ▶ Watch (1:03:01)
Notable Quotes
yeah but AI is really for memes right Jason Haddix · ▶ Watch (2:57)
I don’t know that’s not verified by anybody Jason Haddix · ▶ Watch (28:44)
absolutely would have click that Jason Haddix · ▶ Watch (47:56)
Key Takeaways
- Structured system prompts with identity, weird machine tricks, and research terms beat single-shot queries by 30 to 35%.
- AI agent frameworks build personalized spearphishing emails from open-source data, generating content that fools experienced red teamers.
- Blue team bots for ELK, Splunk, osquery, and Yara remove the query-writing bottleneck from SOC analysts’ daily workflow.
About the Speaker
Jason Haddix, known as jhaddix, is the CEO and Hacker in Charge at Arcanum Information Security, a security assessment and training company. His 20-year career includes roles as CISO at Buddobot and Ubisoft. He has presented at Defcon and teaches a two-day AI for Security training quarterly, covering bot-building methodology for red team, blue team, and leadership functions. His Arcanum cybersecurity bot on the ChatGPT store holds the highest security helper rating on the platform.