AI
AI Adoption Outpacing Security Awareness
Gartner projects 19% annual AI growth reaching a $300 billion market by 2027. Orca’s own scan data shows 39% of Azure organizations have deployed OpenAI, 29% of AWS users run SageMaker notebooks, and 24% of GCP users run Vertex AI. More telling: 56% of those organizations embedded AI into custom applications rather than using off-the-shelf services. The exposure follows adoption. 62% of organizations using AI packages are already vulnerable to flaws native to those packages, and 20% stored OpenAI API keys in insecure locations.
AI Goat: A Hands-On Vulnerable Infrastructure
AI Goat is an open-source project from Orca Security that deploys a deliberately vulnerable AWS environment for practicing machine learning attack techniques. The front end is a toy web store. Behind it sits Amazon SageMaker running models with intentional flaws tied to the OWASP Top 10 Machine Learning Risk list, a framework the OWASP team published in early 2023. Deployment takes one AWS account with admin privileges, a git clone, and terraform apply. The current build ships three challenges at different difficulty levels.
Supply Chain Attack via Metadata Shell Injection
The image similarity model accepts photo uploads for product search. A failed API call in the browser network tab leaks a GitHub repository URL for the preprocessing code. That repo is public. Inside: a process_image function using the Pillow library that reads EXIF metadata and passes it directly to a shell command, returning the output. Embedding ls /home/ec2-user in image metadata returns a directory listing. A second crafted image reads a file named sensitive_data, which contains an S3 bucket name used in the next attack.
Data Poisoning via a Publicly Writable S3 Bucket
The recommendations model draws from a product_ratings.csv stored in the bucket found during the first attack. Anonymous read access works. Anonymous write access also works. The CSV holds user_id, product_id, and ratings columns. Product 2 (the hidden Orca doll) has low ratings; product 3 has fours and fives. Replacing every product 3 row with product 2 and giving all rows a rating of five triggers the retraining pipeline. The SageMaker endpoint shifts from “running” to “updating” and, five minutes later, the Orca doll tops the recommendations.
Output Integrity Bypass via Payload Injection
The comment section runs a filtering model that blocks offensive words, including “pond”. Each request sends author, content, is_offensive, and probability. The is_offensive and probability fields are supposed to be response values from the model, not request inputs. Sending “pond” with is_offensive set to 0 and probability set to 0.8 still fails. Changing the author to “admin” also fails. But sending is_offensive=1 and probability=1 alongside “pond” bypasses the filter. The comment posts. The application appears to cache prediction values and skips re-querying the model when those fields are present in the payload.
Mitigations and the Road Ahead for AI Goat
The talk closes with six defenses for ML deployments. Least-privilege access control keeps models from touching resources they shouldn’t. Input and output validation blocks payload injection attacks. Vulnerability management tracks CVEs in dependencies. Continuous monitoring logs model inputs, outputs, and resource usage to catch anomalies. Sensitive data must stay out of training sets. Developer education, the speakers’ top priority, addresses the human factor. On the roadmap: the remaining seven OWASP machine learning risks and a fully local version without AWS dependency.
Notable Quotes
we have a file called sensitive data uh Shir Sadon · ▶ Watch (16:11)
spoiler it does have some retraining Shir Sadon · ▶ Watch (23:53)
word Pond was being commented Shir Sadon · ▶ Watch (29:53)
Key Takeaways
- AI packages expose 62% of adopting organizations to native vulnerabilities; exposed API keys extend that risk further.
- Publicly writable S3 buckets let attackers poison training data and trigger model retraining without credentials.
- Sending model output fields in the request payload bypasses prediction-based content filters entirely.
About the Speaker(s)
Ofir Yakobi is a Cloud Security Researcher at Orca Security with almost a decade of experience in detecting cybercriminals, malware research, and exposing security issues at high-profile companies. She specializes in breaking and strengthening cloud vendor security.
Shir Sadon is a cloud security and martial arts enthusiast at Orca Security. With a background in endpoint and server security, Shir led research ventures to enhance departmental security posture and now focuses on cloud security and AI.