Third-Party Models as a Code Execution Vector

▶ Watch (02:04)

Hugging Face hosts nearly two million models. When a data scientist downloads one, they risk running malicious code on load or at inference time. The threat comes not from the weights but from the architecture, which defines circuits and relations between those weights. Pickle serializes that architecture, and pickle is executable. TensorFlow embeds Python lambda functions inside JSON files. Even SafeTensors has documented paths back to pickle loading.

The model’s creator controls every output. Any scanner that treats architecture code as mere data misses the root problem.

Why Static Scanners Fail by Design

▶ Watch (06:33)

Static scanners read model file bytes without executing them. They extract the import list and check each entry against a deny list. If os.system appears, the model is flagged. If an import is unknown, Hugging Face’s pickle scan tells the data scientist to decide. That decision point is where the system breaks. Scanners produce three outcomes: safe, unsafe, and “I don’t know.” In the third case, data scientists consult vendors and are routinely told the model is fine.

Every scanner is reactive: it can only block functions already on its list.

Bypass Methods: Wrapper Functions and Bytecode Obfuscation

▶ Watch (09:27)

During the AI-generated wrapper function bypass demo (09:52), an AI agent produced 50 wrapper functions around unsafe calls in two hours. MLflow’s backend_local_run_entrypoint wraps CLI execution, sits on no deny list, and every consulted scanner rated it safe. Thousands of packages update daily. No static list keeps pace.

“to write an AI agent. This is 2025,” — Itay Ravia

The second method uses bytecode. Dill and cloudpickle embed Python bytecode directly in model files. Eight lines split the “os” string across variables, reconstruct it, then call os.system. Bytecode analysis is a generalization of the halting problem: no scanner resolves it in bounded time.

Pickle Opcode Desynchronization

▶ Watch (13:46)

In the pickle opcode desync bypass walkthrough (16:38), pickle scan’s put-mem handler assumes the previous opcode’s argument is what lands in memory. An attacker pushes “os”, calls instantiate builtins.str("os"), then fires put-mem. The scanner records builtins.str; the unpickler records “os”. A later python-import for “system” imports os.system while the scanner sees builtins.str.system. Not on any deny list, it passes. A fourth method, joblib, embeds a runtime-length random byte array mid-file, forcing scanners to stop analysis before the malicious import. One vendor left Ravia’s test model on Hugging Face undecided for over a month.

Dynamic Sandbox Tracing as a Detection Approach

▶ Watch (24:43)

Running model files inside a sandbox and tracing their system calls sidesteps every static limitation. A model loading weights has no reason to spawn processes or make network connections, so the allow list of legitimate operations is short. Every deviation flags automatically, no function name needed.

“running Python is not NP hard and and there’s no” — Itay Ravia

The dynamic scanner caught the MLflow wrapper and the joblib obfuscation without knowing either method in advance. New bypass techniques get detected on first use, and the behavioral signature can then backfill the static deny list.

Q&A

What tool were you using for the static and dynamic scanner screenshots? Ravia clarified it was a Keras model with an embedded lambda function that called CLI commands, not a named tool; the static scanner could only flag the lambda as ambiguous, while the dynamic scanner traced the actual system call. ▶ 31:17

Aren’t you just rediscovering the RCE cat-and-mouse game that antivirus has been playing for years? Ravia said the dynamic approach avoids that arms race because it allow-lists a small set of expected model operations rather than chasing RCE methods; anything outside that set is flagged regardless of technique. ▶ 32:12

Notable Quotes

to write an AI agent. This is 2025, Itay Ravia · ▶ 9:52

that um it’s just impossible to create a Itay Ravia · ▶ 11:26

running Python is not NP hard and and there’s no Itay Ravia · ▶ 29:44

Key Takeaways

  • Pickle, TensorFlow lambdas, and joblib all allow executable code inside model files downloaded from Hugging Face.
  • An AI agent built 50 deny-list-evading wrapper functions in two hours, proving exhaustive static coverage is impossible.
  • Sandboxed execution catches every bypass variant because models have a narrow, predictable set of legitimate system calls.