AI
Why Adversarial Inputs Break AI Systems Silently
Adding imperceptible noise to an image can make a self-driving car misclassify a stop sign. In one study, a plane image fed with adversarial noise was classified as a frog. Humans cannot detect these perturbations, and 72 to 85% of such attacks go undetected. Traditional security tools catch buffer overflows and SQL injections. AI systems fail on inputs that look correct to any human reviewer, which means the threat model has to change.
The Gap Between AI Testing Standards and Real Threats
AI security standards have three gaps. First, a standard built for one AI application may not apply to another. Second, tests are not exhaustive the way traditional software tests can be. Third, testing an individual AI component in isolation is nearly impossible. Most testing happens at the system level. LLMs compound this: attackers can now scale their campaigns without spending more time or effort. The bar to attack drops while the bar to defend stays high.
Building a Threat Library From 14 Sources and 429 Candidates
The team started with a literature review across digital libraries and academic sources. After filtering out frameworks without security/privacy content and those operating at a national rather than application level, 14 sources remained. Unified, those sources produced 429 candidate threats. After removing duplicates, the team had a unique threat list that mapped attack surfaces within AI applications. Ten expert interviews validated the final questions for reliability and clarity before the questionnaire was finalized.
Three Question Tiers That Cover Any AI Application
Guardrail’s 47 questions split into three tiers. Every AI application answers the Baseline set. Systems that update their model incrementally add the Continuous Learning questions. Any application processing user data or PII adds the User Data Interaction set. Each tier subdivides by threat location: data modules, the model itself, output artifacts, and system infrastructure. The project is referenced in the NIST AI Risk Management Framework Playbook, the OWASP Top 10 for LLM Applications, and the OECD catalog of tools for trustworthy AI.
Q&A
How do red teams and SAST tools fit into AI model adversarial testing? The plan is to use existing tools where available and build new ones for gaps, though the adversarial testing track is still early-stage. ▶ Watch (26:23)
Which of the 47 questions surfaces the most risk? The PII-handling questions, starting with what data the application processes and whether that PII is used to update the model. ▶ Watch (27:25)
Notable Quotes
to defend is quite high in this case uh but the uh bar to attack is quite low Dr. Nitish M. Uplavikar · ▶ Watch (5:38)
like just complying to standards does not mean the application uh is secure so Dr. Nitish M. Uplavikar · ▶ Watch (12:49)
Key Takeaways
- Adversarial noise that humans cannot see can force AI misclassification in 72 to 85% of cases.
- AI testing standards lack exhaustiveness and component-level testing, leaving gaps that LLMs help attackers fill.
- Guardrail distilled 429 threats from 14 sources into 47 actionable questions for threat modeling.
- The three question tiers, baseline, continuous learning, and user data interaction, cover any AI application type.
- NIST, OWASP, and OECD have all referenced Guardrail as a resource for AI security practitioners.
About the Speaker(s)
Dr. Nitish M. Uplavikar is a cybersecurity researcher at Comcast’s Security Privacy Innovation Development Engineering and Research (SPIDER) team. His work focuses on security and privacy research within threat modeling, including the development of tools like Guardrail to address real-world AI security problems.