AI
Why Phishing Training Lacks an Evidence Base
Phishing training is nearly universal. It is required for cyber security insurance, considered best practice, and comes in two forms: annual awareness modules and simulated phishing tests with embedded remediation. Most practitioners assume it works. The evidence base says otherwise.
The researchers framed the problem using medicine’s evidence pyramid. Security sits near the bottom, relying on expert opinion and vendor claims. The top — randomized control trials — is where medicine goes before approving a drug. Security almost never gets there. Before spending millions of dollars and millions of hours of employee time, Dameff and his co-researcher wanted to know if the intervention actually moved the needle.
The RCT Design: 19,000 Employees, Five Groups, Eight Months
The study enrolled 19,000-plus employees across three hospitals — doctors, nurses, cafeteria staff, sanitation workers — at UCSD Health. Participants were randomized into five groups: a no-training control, generic static, generic interactive, contextual static, and contextual interactive. Contextual training used the specific lure the employee had just failed, rather than generic phishing advice. Over eight months, the team deployed 10 different lures monthly and measured failure rates, training engagement time, and days since last annual training.
The vendor platform provided no built-in engagement timer, so the researchers hard-coded their own metric into the back end to capture how long each user spent on training.
Lure Content Drives Failure Rates More Than Training Does
Failure rates across the 10 lures ranged from 1.8% to 30%. A plain-text dress-code violation notice hit 27%. A vacation policy change hit 30%. The Outlook password reset sat near the bottom. The high-failure lures had no images or formatting tricks. Content alone drove the gap.
“whoever controls the lures controls the failure rate” (Ariana)
By month eight, over 50% of all users had failed at least one simulation. A security team told to prove training effectiveness can send the low-failure lure. One told to maximize training exposure can flip to the high-failure one. The program’s reported outcome reflects lure selection as much as employee behavior.
Training Produced a 1.7% Improvement — and Users Skipped It
Across all four training groups, embedded phishing simulations improved outcomes by 1.7% over the control. Annual cyber security training showed no detectable signal. Failure rates held flat regardless of how recently an employee had finished the yearly module.
Between 40% and 50% of post-failure training sessions recorded zero seconds of engagement. Users opened the page and closed it before any time registered. The median session ran 0 to 10 seconds. People who spent more than 90 seconds on static training actually performed worse afterward. Interactive training showed a small positive effect, but the subgroup was too small to generalize confidently.
“the average improvement was only 1.7%” (Ariana)
Rethinking Where the Burden Should Fall
The researchers asked whether 1.7% improvement justifies the cost. An LLM takes under a second to rewrite a 2% click-through lure into a 30% one, erasing any training gain. Attackers send as many emails as they want. One user having a bad day is enough.
Their two recommendations: treat security like medical research (measure outcomes, share data, accept peer review) and shift the protection burden from users to systems. Hardware keys and AI-based inbox filtering don’t require employees to pass a test under pressure. Stop letting conflicted vendors be the sole collectors and analysts of whether their own products work.
“Back up your claims with data.” (Christian Dameff)
Notable Quotes
whoever controls the lures controls the failure rate Ariana · ▶ 13:23
the average improvement was only 1.7% Ariana · ▶ 17:55
Back up your claims with data. Christian Dameff · ▶ 25:30
Key Takeaways
- Lure content alone shifted failure rates from 1.8% to 30%, giving lure-selectors direct control over program metrics.
- Across 19,000 employees over eight months, phishing training reduced failures by only 1.7% over an untrained control group.
- 40-50% of post-failure training sessions lasted zero seconds; median engagement was 0-10 seconds across all modalities.