Sidecar Containers: Init Containers in Disguise

▶ Watch (2:58)

Kubernetes 1.29 made sidecar containers official, but the implementation surprises most people. There is no sidecars: field in a pod spec. Instead, you set restartPolicy: Always on an init container. That single flag tells the kubelet to keep the container running past the init phase and alongside your main application containers. Noble described this as syntax he actively dislikes. The design reason remains unclear — both speakers asked the audience and got no answer.

Image SHA Tags: When Humans and Kubernetes Read Different Things

▶ Watch (4:59)

Referencing an image by SHA is best practice because the digest is tied to content. If the content changes, the SHA changes. The trap comes when you pin both a human-readable tag and a SHA in the same image reference. Kubernetes ignores the version tag entirely and uses only the SHA. If a tool like Renovate or Dependabot updates the SHA without also updating the tag, your team reads 1.25.1 but the cluster runs 1.5.2. Noble confirmed he has been caught by this personally.

Pod DNS: Why “Default” Is Not the Default

▶ Watch (7:25)

Pod hostname and subdomain fields exist and are writable, but setting them does not make a pod addressable by name. Pods cannot have DNS records assigned directly, even if a hostname is set. Headless services (services without a cluster IP) work around this by prepending the pod name to the domain. On DNS policy, four values are valid. The actual default is ClusterFirst, which enables cluster service name resolution. The value literally named Default falls back to the node’s resolv.conf and skips cluster DNS entirely.

RBAC Escalation and Admission Policy Blind Spots

▶ Watch (10:00)

RBAC prevents users from granting permissions they do not already hold — unless the escalate verb is present. A role binding that includes escalate on RBAC resources lets the holder create cluster roles with any permissions, regardless of their own. Separately, validating and mutating admission webhooks cannot restrict themselves. Kubernetes source code explicitly skips webhook evaluation for requests that create, edit, or delete webhooks. This prevents a bad actor from locking out future policy changes, but it also means webhook creation is always ungated by admission controls.

Static Manifests and the Token Request Trap

▶ Watch (10:07)

The Token Request API creates JWTs against any service account, including privileged ones like kube-system:node-controller. Those tokens cannot be revoked or rotated. The only remediation is deleting the service account or waiting for expiry. Noble recommended short TTLs — 24 hours or less depending on environment. Separately, kubelet standalone mode runs pods from static manifests on disk with no API server involved. A pod placed in a nonexistent namespace never gets a mirror pod in the API server, so it does not appear in kubectl get pods.

Notable Quotes

I hate this inconsistency. This is like the legacy stuff from the early days of Kubernetes that we now can’t change. Marcus Noble · ▶ 07:07

default means it’s going to fall back to the nodes resolve comp values. Cluster first is going to be the default where you can actually use the cluster DNS and resolve service names. Márk Sági-Kazár · ▶ 09:43

you can hide pods running on worker nodes as static manifests, but they’re not going to show up on when you run cubectl. Sorry, cube cuddle. get pods. Márk Sági-Kazár · ▶ 17:19

once you’ve got this, you’ve then got a JWT all token that cannot be revoked and it cannot be rotated. Marcus Noble · ▶ 10:53

Key Takeaways

  • Sidecar containers are init containers with restartPolicy: Always, added in Kubernetes 1.29.
  • When a SHA and a version tag both appear in an image reference, Kubernetes uses only the SHA.
  • The escalate RBAC verb lets a user grant permissions beyond their own current access level.
  • Admission webhooks cannot govern the creation or deletion of other admission webhooks.
  • Kubelet standalone mode bootstraps the control plane and can hide pods from the API server.

About the Speaker(s)

Marcus Noble is a platform engineer at Monzo, a UK-based bank, focused on Go, Kubernetes, containers, and DevOps. He started as a web developer and JavaScript enthusiast and describes himself as a tinkerer.

Márk Sági-Kazár is a software engineer from Hungary active in open source as both maintainer and contributor. He organizes KCD Budapest and publishes a hands-on course called “Kubernetes the Very Hard Way” on Exuse Labs, covering the inner workings of Kubernetes from the ground up.