AI
Why Confidential Clusters Need Node Attestation
A Kubernetes cluster holding regulated data faces a specific threat: a malicious cloud administrator can access memory, disk, or network traffic at the hypervisor layer. Confidential computing closes that gap by protecting data in use, not just in transit or at rest. Alice Frosi and Jakob Naucke define a “trusted execution cluster” as one where the confidentiality boundary is the cluster itself. Every node must run on confidential hardware: AMD SEV, Intel TDX, or ARM equivalents. The cloud provider sits outside the trust boundary. The cluster administrator sits inside it.
How Remote Attestation Proves a Node Is Trustworthy
Remote attestation uses a trusted hardware key on each machine. At boot, the TPM signs a “quote” containing PCR measurements, which are hashes of the boot chain. The signed quote goes to Trustee, a CNCF project from the Confidential Containers community that acts as both key broker and attestation service. Trustee validates the TPM key, compares the measurements against stored reference values, and returns a signed attestation token. The node uses that token to release a LUKS key and decrypt its root disk. No valid token means no access.
Node Enrollment via Ignition, Clevis, and PCR Identity
When a new node joins, it boots with an Ignition configuration that fetches additional config from a registration server. That config specifies disk encryption using a novel Clevis pin for Trustee. Clevis encrypts the root volume by calling the Trustee attestation client. Node identity comes from init data: a UUID written immutably into PCR8. Once that PCR is extended, the node cannot impersonate a different registered node. The registration server returns the Clevis config; from that point, both new and existing nodes follow the same boot path through attestation to decrypt their disk.
Managing Reference Values Through Node Updates
The attestation policy checks specific PCR banks. PCR4 measures the bootloader; PCR11 measures the UKI. When a kernel or bootloader update lands, reference values change for the next two boots because both components update separately. The TEC operator handles this with a “compute-pcrs” tool. A cluster admin creates an ApprovedImage custom resource pointing to a bootc image (a bootable container that includes kernel, init system, and applications). The operator computes expected PCR values for that image and stores them in Trustee. Every reference value set is keyed to the node’s UUID.
Memory Exposure: Standard VMs vs Confidential VMs
The demo runs two VMs using KubeVirt: one standard, one confidential (AMD SEV-SNP). Both run Kubernetes and host a Python app. After sending a secret via curl POST, Alice creates a memory dump and scans it with a Kubernetes job. On the standard VM, the secret appears in plaintext. On the confidential VM, the memory is encrypted and the scan finds nothing. The encrypted root disk uses a LUKS key retrieved from Trustee after attestation. Secure boot is disabled for now but planned for a future release.
What’s Next for Trusted Execution Clusters
TEC currently targets Azure, with GCP and AWS planned. The next goal is attestation-gated cluster join: protecting the kubeconfig and ignition config as secrets, so a node must pass attestation before its CSR is accepted. Right now, attestation depends on an external “cluster zero” that must be independently trusted, which creates a bootstrap problem. Attesting the control plane itself comes next. The operator, registration server, and machine components are all written in Rust, using the kube-rs library for Kubernetes API access.
Q&A
Is Trustee essentially a Tang server that validates TPM measurements and returns a LUKS key? Yes, Trustee acts as a key broker that releases secrets only when attestation passes. ▶ 26:05
How does the TPM key used to sign quotes get established on the machine? The key resides inside the TPM; Trustee retrieves the machine’s public key and verifies the quote signature against it. ▶ 27:10
Can a cluster fully trust a confidential VM if cloud-provider firmware also appears in the measurements? Cloud-contributed components must be accepted on trust; the operator runs its own attestation but cannot verify opaque firmware provided by the cloud. ▶ 28:46
Where should the remote attestation server be hosted? It must run in an independently trusted “cluster zero” hosted on-premises or at a trusted entity; the team does not yet provide a managed option. ▶ 31:08
Notable Quotes
the memories is basically in clear. Alice Frosi · ▶ 22:14
So nothing was found. Alice Frosi · ▶ 23:11
Key Takeaways
- TPM-signed PCR measurements gate LUKS key release; nodes with altered boot chains cannot decrypt their disk.
- A node’s UUID stored immutably in PCR8 prevents it from impersonating other registered nodes.
- Standard VMs expose secrets in memory dumps; AMD SEV-SNP machines do not.
- Kernel or bootloader updates change TPM reference values for two subsequent boots; the TEC operator manages this automatically.
About the Speakers
Alice Frosi is a Principal Software Engineer at Red Hat, working on KubeVirt, virtualization, and containers. Her primary focus is storage, though she has fun trying all possible combinations of containers and VMs.
Jakob Naucke is a Software Engineer at Red Hat’s Virtualization Group and a major contributor to the Confidential Clusters project. Before his MSc degree, he worked at IBM in the Kata Containers and Confidential Containers community.