AI
New ML Cloud Providers Give Root Access to Poorly Hardened Hardware
New GPU cloud providers offer ML compute at a tenth of AWS or Azure prices, selling bare metal: direct hardware access with no hypervisor between customer and host. Vendors advertise physical isolation and improved security. Customers get root on actual hardware by design. That exposes UEFI/BIOS firmware, onboard PCI devices, the baseboard management controller, and the data center network. Any firmware or hardware attack that normally requires physical access becomes possible when root is already granted. Established clouds spent over a decade hardening these surfaces. Newer providers have not.
UEFI HTTP Boot Creates Disk-Free Persistence Across Customer Tenancies
MAAS (Canonical’s Metal as a Service) provisions servers via PXE boot over DHCP, a protocol nearly two decades old. An attacker with root can add a UEFI boot option pointing to a remote EFI image via HTTP boot, a feature in EDK2 firmware for over a decade. UEFI authenticated variables let the attacker sign-protect this entry, making it hard to remove without firmware-level access. Bill also showed AMI BIOS option offsets can be parsed from undocumented firmware structures, letting an attacker silently downgrade security features without console access.
BIOS Downgrade Attacks Bypass Boot Guard and AMD ROM Armor
Intel Boot Guard and AMD ROM Armor protect firmware integrity by enforcing code signatures, but neither prevents rollback to older, signed-but-vulnerable versions. Intel has secure version numbers that could prevent downgrade attacks, but no major OEM uses them. Bill found he could flash an older vulnerable BIOS version, exploit its bugs to escalate from OS to boot environment, and still pass Boot Guard because the code was legitimately signed. Chipsec flags writable BIOS regions and manufacturing mode on Intel. On AMD, Platbox from IOActive found at least one major issue on nearly every machine Bill tested.
BMC Signature Checks Are Often Client-Side Theater
The BMC is a dedicated chip that runs independent of the host OS, enabling full remote management even when the server is powered off. Bill found it reliable for backdooring. Proprietary customizations on base firmware (like ASUS code on MegaRAC) are less audited and contain more bugs. He found undocumented IPMI commands that reset BMCs to factory defaults with known credentials. When Gigaflash rejected his modified BMC image, a different vendor’s tool for the same AST2500 chip flashed it without complaint. The signature check was client-side only.
Unencrypted MAAS Provisioning Enables Cross-Tenant Code Execution
Bare metal access removes the need to phish employees. Bill found promiscuous mode captured other customers’ traffic on several providers without any ARP manipulation. Where basic switch segmentation existed, ARP spoofing worked immediately. MAAS serves boot images and provisioning scripts over unencrypted HTTP and DHCP. An attacker on the same segment can respond to a DHCP Discover with a malicious PXE URL, injecting their own EFI image at boot time for code execution. MAAS also exposes a Squid proxy by default; Bill used HTTP CONNECT through it to port-scan the internal network.
RunPod Customers Exposed Through a Provider’s Provider
RunPod offers GPU Docker containers on a secured cloud of vetted data centers, with no bare metal rentals. Traceroutes from RunPod servers led Bill to Mass Compute, a bare metal provider offering hourly rentals. ARP spoofing the MAAS gateway on Mass Compute’s network exposed unencrypted traffic from RunPod’s customers: Calteex ML training runs and inference service logs including container names. RunPod had no bare metal exposure but inherited it from their provider. They responded quickly and isolated customer networks before the talk. Most other providers did not.
Notable Quotes
don’t believe in confidential compute. Bill Demirkapi · ▶ 44:38
just complete BS at best a deterrent. Bill Demirkapi · ▶ 44:43
it is prime for a back door. Bill Demirkapi · ▶ 23:05
Key Takeaways
- PXE/HTTP boot persistence via UEFI authenticated variables survives server decommissioning without firmware reflash.
- BIOS downgrade attacks bypass Boot Guard and AMD ROM Armor because old signed code is still legitimate.
- BMC signature verification is often client-side only; a different vendor’s flashing tool circumvents it.
- Promiscuous mode and ARP spoofing expose other customers’ traffic on most ML cloud providers tested.
- Container cloud providers inherit bare metal network vulnerabilities from the providers they resell.
About the Speaker(s)
Bill Demirkapi is a security researcher focused on finding bugs at scale. His interests span reverse engineering and vulnerability research, from low-level memory corruption to systemic flaws with catastrophic consequences. He began security research in high school and has since published at DEF CON and Black Hat USA. He follows the motto “break anything and everything”.