AI
The Bootstrap Token Problem: Ten Years Without a Standard
Every panelist admitted the same failure: ten years of Kubernetes and no agreed standard for secure node joining. Justin Santa Barbara framed it plainly, noting that what kOps does on AWS differs from GCP and Azure, and no Kubernetes standard checks whether any of those implementations are correct. The join token was introduced as a convenience for better user experience, not as a security primitive. Across kOps, Cluster API, and Karpenter, nodes still rely on shared secrets distributed through mechanisms like instance metadata.
How Metadata Service Abuse Becomes Cluster Takeover
The recurring CVE pattern across cloud providers is the same: too much information placed in the instance metadata service. A routine container escape escalates to full cluster ownership because the metadata carries enough credentials to take over. On IBM Cloud, Josie Pfeiffer described arbitrary code execution as root on new nodes when user data scripts were not locked down. Justin Santa Barbara said he has spent 10 years whittling down node permissions and is very much not there yet.
What Node Attestation Actually Means
Node attestation means a node can self-identify to another resource and that claim can be trusted. The panel split this into two layers. Hardware attestation asks whether the instance is the one requested and belongs to the expected cloud. Software attestation asks whether the OS and kubelet are what the operator expects. Cloud-init and Ignition cover boot mechanics but neither layer has a standardized attestation handshake. Hybrid clusters, where a managed control plane accepts nodes from external infrastructure, make the gap worse.
The Fragmentation Across Clouds
GCP ships a virtual TPM. Azure offers attested metadata signed with the cloud provider certificate. Smaller VPS providers give operators nothing beyond an IP address and an instance metadata endpoint added under pressure. Ciprian Hacman noted that attested metadata validation still varies per cloud: you need a public key or certificate from the provider to verify what arrives. There is no conformance program, no sig-level audit, and no shared implementation between kOps, Cluster API, and Karpenter.
Proposals From the Floor: TPM, Device Certificates, and Short-Lived Tokens
Audience members offered three directions. One proposed virtual smart cards so the platform proves identity by issuing a certificate directly, keeping secrets off the machine entirely. A second pointed to the ACME DeviceAttest01 challenge as a possible standard entry point for device-bound client certificates. A third argued that Cluster API node claims, combined with short join windows, already reduce the attack surface because each claim maps one-to-one to an expected instance. The panel agreed that a core implementation with thin cloud-provider wrappers would reduce sprawl.
Q&A
Has kOps actually solved this for AWS? Justin confirmed kOps uses an additional handshake and does not use the join token, but the implementation is unaudited by sig-auth, differs from GCP and Azure, and is not shared with other install tooling. ▶ 23:45
Can TPM provisioning work for physical hardware, not just VMs? One panelist argued yes, because hardware providers automate provisioning and can install certificates in the TPM before handing over the machine. Cyprian pushed back, noting that many mid-tier clouds provide only an instance and an IP. ▶ 24:59
Notable Quotes
They’re all basically the same vulnerability. The we put too much uh information into the metadata service, the instance metadata service, and that information is sufficient to enable relatively what would normally be a simple container escape, uh can be escalated into much more severe things like a cluster ownership type thing. Justin Santa Barbara · ▶ 03:18
I I would say that it is not even good enough across the board and and you should you can maybe make it acceptably safe in certain circumstances when you’re minting a join token for each new machine that joins the cluster but that is a relatively rare configuration. Justin Santa Barbara · ▶ 11:45
the only people that have really audited that code are you know a handful of of kind people. Justin Santa Barbara · ▶ 24:16
whatever solution we come up with uh it should be very low effort for the consumer and I think you know we chose something that was low effort for the implement i.e us in the form of those bootstrap tokens. Jack Francis · ▶ 30:17
Key Takeaways
- Bootstrap join tokens are not a security primitive; they were added for convenience.
- No Kubernetes sig audits node-joining implementations across kOps, Cluster API, or Karpenter.
- TPM-backed attestation with virtualized secure elements is the panel’s agreed target state.
- Many mid-tier clouds supply only an IP address, making TPM-based solutions unavailable today.
- A core shared implementation with thin cloud wrappers would reduce fragmentation and audit burden.
About the Speaker(s)
Michael McCune is a software developer at Red Hat building open source infrastructure for cloud platforms, with prior work on OpenStack.
Justin Santa Barbara has contributed to Kubernetes since 2014 at Google, originally as the primary maintainer of Kubernetes AWS support and founder of the kOps project.
Ciprian Hacman is a software engineer at Microsoft specializing in platform engineering and is the current lead of the etcd-operator working group.
Jack Francis is a Principal Software Engineering Lead at Microsoft working on open source Kubernetes from Portland, Oregon.
Josephine Pfeiffer is a Senior Cloud Native Consultant at Red Hat focused on infrastructure and security, including confidential containers and the Karpenter implementation for IBM Cloud.