AI
The Security Gap in “ifNotPresent” Image Pulls
Pod A pulls a private image using secret-backed credentials. The kubelet fetches it and the container runtime stores it locally. Pod B then requests the same image with ifNotPresent policy but provides no credentials. The kubelet finds the image cached locally and runs it. Pod B just accessed a private image with no credentials. The existing mitigation, the AlwaysPullImages admission controller, is disabled by default. When enabled, it forces a registry call on every pod start, adding network overhead and exposing credentials unnecessarily.
A Persistent Credential Store in the Kubelet
Láznička’s fix: a persistent store that records what credentials were used to pull each image. The store uses two record types. A “pulling” record is written before the pull starts. A “pulled” record is written after the image arrives. The split matters because the kubelet and container runtime run asynchronously. If the kubelet restarts mid-pull, the pulling record captures the in-flight state. The design also preserves support for pre-loaded images, such as those KIND places on nodes during cluster setup, which arrive outside the kubelet.
Four Verification Policies for Image Credentials
The credential store adds a new kubelet configuration field, imagePullCredentialsVerification, with four policies. AlwaysVerify requires a credential record before any cached image runs. NeverVerifyPreloadedImages (the default) skips checks for images loaded outside the kubelet. NeverVerifyAllowlisted skips checks for a named set of images. NeverVerify disables all verification. A second field, preloadedImagesVerificationAllowlist, pairs with the third option. Naming these took, by Láznička’s count, five of the six months the feature required to develop.
Conditional Authorization: The “Maybe” Mode
Current RBAC can’t scope a create permission to a specific field value. The workaround requires granting broad create access and adding deny rules to exclude unwanted cases. Käldström’s KEP adds a third authorizer response: conditional. When the authorizer can evaluate some conditions but not others (the object isn’t decoded during authorization), it returns conditions with the request. These travel into admission, where they resolve against the actual object. CEL conditions evaluate inside the API server using the same code as ValidatingAdmissionPolicy, so no external webhook call is needed.
Q&A
If a credential identity later loses access to the private registry, does the credential store detect that? The store records secret metadata and a hash of secret content; once an identity pulled an image, the stored record stays valid even if registry access is later revoked. ▶ 27:18
Notable Quotes
we are just two silly men in funny hats. Stanislav Láznička · ▶ 0:10
you don’t have those guarantees anymore. Lucas Käldström · ▶ 16:01
guess we can go home now. Lucas Käldström · ▶ 17:33
Key Takeaways
- The
ifNotPresentpull policy lets pods run cached private images without any credentials. - A new kubelet credential store maps each image to the credentials that pulled it, persisted across restarts.
- Conditional authorization adds a maybe mode to Kubernetes, enabling CEL-based field-level permissions resolved at admission time.
About the Speaker(s)
Stanislav Láznička is a Principal Software Engineer at Microsoft with deep experience in authentication, authorization, and certificates in Kubernetes.
Lucas Käldström is a Staff Software Engineer at Upbound and a Kubernetes contributor for ten years. He co-led SIG Cluster Lifecycle, co-created kubeadm, ported Kubernetes to ARM, and was named Top CNCF Ambassador in 2017. He served as a CNCF Ambassador for seven years.