Why weights_only Fails as a Security Boundary

▶ Watch (2:07)

PyTorch’s early model loading relied on Python’s pickle library. The Python documentation flags pickle as unsafe with untrusted data. In 2021, a GitHub issue called for a fix. PyTorch responded in 2022 with a weights_only=True parameter for torch.load. The parameter restricts pickle’s dangerous opcodes using a whitelist and blacklist. The ML ecosystem adopted it immediately. When other projects loaded models unsafely, the standard fix was to add weights_only=True. Researchers at Alibaba Cloud asked whether that trust was deserved.

TorchScript’s Deserialization Pipeline

▶ Watch (8:00)

TorchScript lets PyTorch models run without a Python interpreter, targeting C++ and mobile deployments. Python code is compiled to an internal IR graph, then saved as a .pt file. That file is a ZIP archive with three parts: data.pkl (a pickle blob holding model metadata), constants.pkl (tensor constants), and a code/ directory with the compiled source. Loading calls torch.jit.load, which unpickles all three. During data.pkl deserialization, the loader reads source code from code/, compiles it, and reconstructs the full scripted model in memory.

Over 2,000 Operators, Two of Them Dangerous

▶ Watch (21:00)

TorchScript’s execution engine uses an OP_CALL opcode that dispatches into a global operator table. The researchers patched PyTorch to enumerate every registered operator and found over 2,000. Auditing that list turned up two: save and from_file. The save operator writes a model value to a caller-specified path. The from_file operator reads a file and returns its bytes as a tensor. Both are registered as built-ins and callable from any TorchScript function, inside the layer that weights_only=True is supposed to protect.

Calling Dangerous Built-ins from Inside a Model

▶ Watch (22:02)

To call these operators from inside a malicious model required solving two problems. First, the operators are registered under renamed prefixes, so the researchers traced the built-in registration process to map each function to its callable name. Second, TorchScript uses a lazy export strategy: a method compiles only if called from an already-exported function. Renaming a malicious payload to items and calling it from forward caused vLLM to invoke it automatically during attribute lookup. The resulting exploit wrote an arbitrary file to disk without touching pickle.

Downstream Impact: vLLM and Hugging Face Transformers

▶ Watch (26:48)

A GitHub search revealed thousands of projects using weights_only=True as their only defense. The researchers tested two: vLLM 0.7.3 and Hugging Face Transformers 4.51.3. vLLM had its own CVE for loading without the parameter. Maintainers patched it by adding weights_only=True, but hardcoded the PyTorch version in requirements.txt. Users installing vLLM could not pull in the upstream PyTorch fix. Hugging Face Transformers followed the same pattern. Both teams acknowledged the reports. Hugging Face’s fix gates loading on PyTorch version 2.6 or later.

Defense: Update PyTorch and Switch Model Formats

▶ Watch (34:01)

PyTorch 2.6 patches the TorchScript operator issue by blocking torch.save and torch.from_file when weights_only is active. Anyone still on an older version should update. The researchers recommend using SafeTensors as the model format of choice: it avoids both pickle and TorchScript entirely. Model hosting platforms should scan uploads for malicious payloads. Users who must load third-party models should do so inside a sandbox. The core finding: weights_only=True blocked pickle-based attacks but never covered the TorchScript engine sitting in the same loading path.

Notable Quotes

can we blend in trust and security of owning Ji’an “azraelxuemo” Zhou · ▶ 4:26

because VM has hardcoded the version. So Ji’an “azraelxuemo” Zhou · ▶ 28:26

means we succeed. And this is very Ji’an “azraelxuemo” Zhou · ▶ 30:44

Key Takeaways

  • weights_only=True blocks pickle opcodes but leaves TorchScript’s operator table reachable, creating a second attack path.
  • Two TorchScript built-in operators, torch.save and torch.from_file, grant arbitrary file read and write from inside a model.
  • vLLM hardcoded its PyTorch dependency, preventing users from applying the upstream patch after the CVE was disclosed.
  • PyTorch 2.6 fixes the issue; switching to SafeTensors eliminates the attack surface entirely.
  • Loading untrusted models in a sandbox remains the safest practice until the ML model ecosystem fully adopts secure formats.

About the Speaker(s)

Ji’an Zhou is a Security Engineer at Alibaba Cloud focused on Java security and cloud-native security. His research has led to security improvements at Google, Amazon, Cloudera, IBM, Microsoft, and Oracle. He has spoken previously at Black Hat, Zer0Con, and Off-by-One Con.

Lishuo Song is a Security Engineer at Alibaba Cloud specializing in browser security. He has discovered multiple security bugs in Google Chrome.