AI
AI Infrastructure as an Attack Surface
Every AI workload runs on an NVIDIA GPU. Vector databases like Redis and PGVector, training frameworks like PyTorch and TensorFlow, inference servers like Ollama and NVIDIA Triton — they all share one hardware dependency. The Wiz research team had already found critical RCE bugs in many of these frameworks. Then they asked: what sits beneath all of them?
NVIDIA’s GitHub has nearly 600 open source repositories. That is just the public layer. The researchers zeroed in on a single library powering GPU access across every cloud and SaaS AI service. One bug there meant one bug everywhere.
How NVIDIA Container Toolkit Mounts GPU Resources
When a user runs docker run --gpus all, the Docker CLI generates an OCI spec and hands it to the NVIDIA container runtime. That binary modifies the spec to enable GPU access, then calls runc. Runc initializes the container, then invokes the NVIDIA container runtime hook — added by the previous step — which performs the actual filesystem modifications before the container’s entry point executes.
Those modifications are bind mounts. NVIDIA container toolkit takes files from a compat directory inside the container and mounts them into /usr/lib. Both source and destination are inside the container — not from the host. Running mount inside an NVIDIA-backed container shows dozens of shared objects mounted this way, mostly driver libraries with names like libNVIDIA.so.
TOCTOU Vulnerability and Container Escape Exploit
The security check (do_path_resolve) runs once, before any mounts execute. Riancho’s exploit places two plain folders — libnvidia6 and libnvidia7 — inside compat/, which pass the check. When the first mount writes through a usr/lib → compat symlink, it overwrites compat/’s contents. By the second mount, libnvidia7 now points to the host root. The host filesystem lands inside the container.
“Your image is the exploit.” — Andres Riancho
Host /etc/shadow read from inside container (18:57) shows the read via /usr/lib/libnvidia8, then full host process access through docker.sock. No memory corruption. 100% reliable if NVIDIA container toolkit was present.
“this was a dream vulnerability. It was a single vulnerability affecting the entire cloud ecosystem.” — Andres Riancho
Real-World Exploitation: Replicate and DigitalOcean
On Replicate, scanning /proc from the escaped container immediately exposed a Redis password. That Redis instance is Replicate’s shared task queue — every customer prompt passes through it. Replicate Redis task queue takeover (26:04) The researchers had proven control of this queue in prior work: read any prompt, return any answer, for any model on the platform.
On DigitalOcean’s Paperspace service, the escaped node held kubelet credentials. Those credentials revealed a cluster of over 700 nodes, shared across all customers. DigitalOcean 700-node Kubernetes cluster (30:37) Testing only against their own secondary accounts, the researchers confirmed access to pod contents, running models, source code, and image-pull secrets — credentials pointing back into customers’ private registries.
Disclosure, Defenses, and a Second Vuln Teaser
NVIDIA assigned a CVE and patched the toolkit in version 1.16.2. All vendors were notified; Replicate and DigitalOcean have since hardened their environments.
“AI security is first and foremost infrastructure security” — Hili Ben Sasson
“containers and containerization should never act as a sole security barrier when separating between environments” — Hili Ben Sasson
Three defenses follow from this research: treat every AI infrastructure dependency as attack surface, not just application code; add virtualization or a safe runtime like gVisor beneath containers; keep critical libraries like NVIDIA Container Toolkit patched. On the way out, Ben Sasson previewed a second CVE in the same toolkit, found months later. Three lines of code. Simpler exploit, wider impact. The container runtime is a persistent attack vector, not a one-time finding.
Notable Quotes
Your image is the exploit. Andres Riancho · ▶ 17:42
this was a dream vulnerability. It was a single vulnerability affecting the entire cloud ecosystem. Andres Riancho · ▶ 21:24
AI security is first and foremost infrastructure security Hili Ben Sasson · ▶ 33:51
containers and containerization should never act as a sole security barrier when separating between environments Hili Ben Sasson · ▶ 34:18
Key Takeaways
- Treat NVIDIA Container Toolkit and every AI infrastructure dependency as part of your attack surface, not just your application code.
- Container isolation is not a trust boundary — always layer virtualization or safe runtimes like gVisor beneath it.
- Exploit vulnerabilities in your own test environment before disclosing; understanding real impact strengthens the report and drives faster fixes.