AI
The Limits of Pull-Time Verification
The Notary Project today signs at the image manifest level. The signing phase produces a cryptographic signature binding the signer’s identity to the manifest digest. That signature gets stored as a separate OCI referral artifact alongside the image in your registry. At pull time, notation verifies the signature confirms the image came from a trusted signer and was not modified in distribution. Once the pull finishes, individual image layers are cached on disk. Those cached layers are not continuously re-verified after that point.
Post-Pull Tampering: A Concrete Attack Path
An attacker with host write access can exploit the post-pull window. After notation verifies and the image is pulled, individual layers persist on disk as cached blobs. The attacker modifies one of those cached layers by injecting a malicious binary. When the container runs or restarts, the modified layer mounts directly from disk. No re-verification happens at that point. The malicious binary executes even though the image was authenticated during distribution. In multi-tenant environments, shared layers across multiple containers make this exposure wider.
DM-Verity: Block-Level Read Protection
DM-verity is a Linux kernel feature providing block-level read-only protection. It builds a Merkle tree of cryptographic hashes over a data partition. A container image layer gets partitioned into fixed-size blocks. Each block is hashed, and those hashes are recursively combined until a single root hash remains. That root hash uniquely represents the layer’s entire contents. Changing even one bit changes the root hash. At runtime, the kernel checks every block read against the expected root hash for that layer. A mismatch blocks the read.
Per-Layer Signing at Build Time
For each OCI image layer, the build step generates an EROFS image. EROFS stores data as addressable blocks compatible with DM-verity. A DM-verity Merkle tree then produces a unique root hash for that layer. The root hash gets signed with the signer’s private key. The signature and certificate are packaged into a PKCS7 envelope, a format the Linux kernel can consume directly. That root hash and its PKCS7 envelope get pushed to the OCI registry as referral artifacts attached to the image, alongside the existing manifest signature.
Runtime Verification: Two Checks Before Execution
The extended EROFS containerd snapshot detects the new OCI artifacts at pull time. The container runtime sends the PKCS7 envelope to the kernel. The kernel extracts the certificate and validates it against a trusted certificate authority in the kernel keyring. It then checks the root hash signature using the public key in that certificate. Two separate checks: certificate validity, then signature authenticity. Once both pass, the runtime configures DM-verity with the verified root hash for each layer. Every block read during container execution is then checked against that hash.
Kernel-Level Tamper Prevention and Low Overhead
DM-verity runs in the Linux kernel, not user space. It is already deployed in Chrome OS and Android, running on billions of devices. It hashes only blocks actually read during execution and caches verified blocks, so repeated reads do not pay the hash cost twice. Once a container starts with DM-verity configured, it cannot be disabled on that running container. Even an attacker with root access can modify cached files on disk. DM-verity detects the tampered block and blocks the read before execution. The solution adds per-layer, block-level integrity on top of manifest signing.
Q&A
How does this plug into a Kubernetes cluster, and does it require a specific runtime? The solution uses the EROFS containerd snapshot; you configure containerd to use EROFS. A DM-verity PR for containerd was nearly approved and expected in an upcoming release. ▶ 25:46
If images are owned by root on the host, can an attacker with root access bypass DM-verity? An attacker with root access can modify cached files on disk, but cannot misconfigure DM-verity once the container has already started. Certificate authentication against the kernel keyring runs before execution, and DM-verity enforcement cannot be disabled on a running container. ▶ 26:46
Notable Quotes
those cached layers are not continuously Flora Taagen · ▶ 3:27
detected that block zero is corrupted. Dallas Delaney · ▶ 19:57
And containers can’t opt out. So once Dallas Delaney · ▶ 24:36
Key Takeaways
- Manifest-level signing leaves cached image layers unprotected after pull time.
- DM-verity root hashes, signed with PKCS7 envelopes, close that post-pull gap.
- Once DM-verity starts on a container, tampered blocks are blocked at the kernel read level.
About the Speaker(s)
Toddy Mladenov brings over 25 years of experience in software engineering, consulting, and product management at Microsoft, T-Mobile, and SAP. He joined the Azure team 14 years ago and has focused on large-scale cloud implementations since.
Flora Taagen is a Product Manager on the Azure Linux team at Microsoft. She leads development and adoption of Azure Linux as a platform for Kubernetes workloads on AKS.
Dallas Delaney is a senior engineer on the Azure Linux team at Microsoft. He contributes to container host OS solutions for Azure Kubernetes Service, most recently the Azure Linux OS Guard product.