AI
Why General Models Fail at Multi-Stage Security Tasks
Frontier models are good at atomic tasks. Point a general model at a web app and it will spot an LFI or a cross-site scripting flaw and run an exploit. Chain those vulnerabilities together across multiple steps and the same model falls apart. Brown put it plainly at [un]prompted 2026: the GPT-class benchmarks confirm it, and so does the security community’s own Cybench, CVEbench, and Xbal validation scores. Multi-stage attacks require conjoining different vulnerabilities in sequence. That is a long-horizon agent problem, and general base models, trained on everything from cat-and-dog Q&A to software development, are not optimized for it. Post-training on domain-specific trajectories is the gap-closer.
Open Trajectory Gym: A Practitioner-Grade Post-Training Framework
Brown released OpenTrajectoryGym an hour before the talk. Building a post-training pipeline for one security agent normally means wiring together 36+ libraries and a distributed RL cluster, then debugging whether a failure is the algorithm, the infrastructure, or the data. OpenTrajectoryGym collapses that into one loop. Bring your own model, benchmark, and reward functions. The framework handles the rest.
“I’m not aware of any other framework that does that in the open source let alone in security.” — Aaron Brown
The target user is an ML engineer, not a PhD lab. LangChain, AutoGen, and Strand agents all plug in. The output is a model already aligned to the correct function-call format before it sees a live target.
Three-Stage Training Pipeline: SFT, Online RL, and Prompt Evolution
SFT comes first. The model learns tool-calling format and pentest terminology from synthetic traces before any RL runs. Brown generates those by giving a teacher model a world manifest (network topology, verifiable end reward) to produce labeled trajectories.
Online RL runs eight agents against the same Cybench task in parallel (15:05). Each trace is scored with composite rewards: binary flag capture plus temporal decay and uniqueness penalties. The trainer picks the best trace by group mean and distills it into model weights. After training, a genetic prompt optimizer rewrites the system prompt using past traces, no weight updates required.
Results: Lifting a 27B Model from 12.5% to 35% Solve Rate
Brown ran the two-stage pipeline on Qwen3.5-27B — a dense model released two days before the talk. Baseline solve rate on Cybench: 12.5%. After SFT plus online RL, it reached 35%, nearly tripling the score in under a day on a side-project GPU budget. In the Qwen3.5-27B crypto CTF case study (18:28), the base model hit an AES-variant challenge with a shuffled nonlinear layer, failed after 25 turns, and quit. The tuned model solved it in fewer turns.
“This is not a talk about this is the best model. This is a framework a recipe.” — Aaron Brown
Practical Guidance: Dense Models, Progressive Scaling, and Data Requirements
Start with a dense model. MoE architectures have no open-source fix for weight synchronization across expert nodes during RL, so they break the training loop. Ramp challenge difficulty from easy to hard the way a human would, not straight to expert-level tasks. On memory: a 27B model loaded at lower quantization runs ~60 GB of VRAM, but fill the context window to 128K tokens and that doubles or triples.
“Long horizons are expensive. It’s a cool sounding name, but what does that mean? It means you need very large key value caches and memory.” — Aaron Brown
Brown’s results used only 285 open-source SFT traces. With a full run, budget 20–30k traces for the supervised fine-tuning stage, then add more RL challenges to improve generalization.
Q&A
Does training on CTF-style tasks generalize to finding bugs in real-world applications where you don’t know if a bug exists? Aaron dropped the CTF framing entirely: any task works if it has a verifiable reward signal at the end, and generalization requires better RL environments — he recommends starting with binary rewards and moving toward composite/progressive rewards. ▶ 22:13
From a compute and scalability perspective, which is better: Unsloth or VERL for post-training? Aaron said Unsloth and VERL solve different problems (SFT vs. test-time scaling), neither fully supports security post-training as of the talk, and he used base TRL directly since both tools are built on top of it. ▶ 24:24
Notable Quotes
There needs to be a verifiable reward at the end. Aaron Brown · ▶ 22:48
I’m not aware of any other framework that does that in the open source let alone in security. Aaron Brown · ▶ 8:48
Long horizons are expensive. It’s a cool sounding name, but what does that mean? It means you need very large key value caches and memory. Aaron Brown · ▶ 20:26
This is not a talk about this is the best model. This is a framework a recipe. Aaron Brown · ▶ 17:41
Key Takeaways
- Post-train open-weight models on domain trajectories — general models cannot reliably chain multi-step exploits.
- Use composite reward functions with temporal decay; binary CTF rewards alone produce noisy, non-generalizing agents.
- Start with a dense model, ramp challenge difficulty progressively, and budget 20–30k SFT traces before adding RL.