Private CA Distribution at Kubernetes Scale

▶ Watch (1:12)

LY Corporation operates over 1,000 single-tenant Kubernetes clusters across on-premises data centers in Japan. The platform uses OpenStack for VM management and Ascend, a CNCF sandbox project for service-to-service authentication. A private PKI issues certificates for internal services. External users cannot trust these certificates. Internal users can, because the private CA is pre-installed on company laptops and servers. But at container scale, distributing those private CA certificates is not straightforward.

Why Three Existing Approaches Fall Short

▶ Watch (3:23)

Three approaches exist for injecting private CA certificates into containers. Dockerfile multi-stage builds bundle public and private certificates but require users to maintain image repositories and CI pipelines for every base image. Init containers share a certificate bundle via emptyDir volumes but need complex pod templates or a mutating webhook. Mounting static files is cleaner, but trust store paths differ across Linux distributions. Go uses the OS store, Java maintains its own, and Python and Node.js require a separate environment variable.

How NRI Enables OS- and Language-Aware Injection

▶ Watch (6:59)

NRI (Node Resource Interface) is a plugin API for high-level container runtimes. Plugins run as standalone processes, communicate via Unix domain sockets, and can intercept lifecycle events such as container creation and startup. They mutate container specs: environment variables, mounts, devices, and arguments. NRI ships enabled by default in containerd 2.0 and works with CRI-O. Tsuchiya and Kubo used its create-container hook as the entry point for OS- and language-aware CA injection during container startup.

The Three-Step NRI Implementation

▶ Watch (12:25)

The plugin runs in three stages. The NRI program subscribes to create-container events, checks pod annotations to decide whether injection is enabled, then registers an OCI hook and mounts an env modifier as the container entry point. The OCI hook fires during create-runtime. It reads /etc/os-release and checks for executables like the node command to detect the container environment, then writes CA certificates to the correct locations. Finally, the env modifier sets environment variables such as NODE_EXTRA_CA_CERTS for Node.js, then calls execve to hand off to the application.

What the Approach Trades Off and What’s Next

▶ Watch (15:56)

Compared to existing methods, the NRI plugin keeps pod templates simple: one annotation enables injection without Dockerfiles, init containers, or complex webhooks. Startup delay stays short. The tradeoff is node-level privileges because the OCI hook writes directly to the container filesystem. The team plans a sandbox mechanism to restrict that file access. Two known gaps remain: kubectl exec sessions do not inherit injected environment variables, and read-only container filesystems are not yet supported. Java support is in progress. Integration with ClusterTrustBundle (KEP 3257, from Kubernetes 1.33) is planned.

Q&A

Does the plugin currently support Java, and what runtimes are planned? Java is not yet implemented because it has its own trust store requiring a different import approach. The team is working on it and welcomes requests. ▶ 19:01

Can the plugin replace all embedded CA certificates rather than just appending new ones? Not yet implemented. Options to enable a replacement mode are being planned. ▶ 20:50

Could the containerd API be used to fetch the rootFS path directly, removing the OCI hook? The NRI create-container event does not expose the container ID yet, making a direct containerd API lookup impractical. Staying compatible with CRI-O also motivates the current multi-step design. ▶ 22:06

Notable Quotes

We operate more than 1,000 kubernetes Tsuzuki Tsuchiya · ▶ 01:43

It suggests that the NRI can be a useful Tsuzuki Tsuchiya · ▶ 18:19

we are going to support uh Java Tsuzuki Tsuchiya · ▶ 20:07

Key Takeaways

  • NRI hooks into the create-container event to handle OS and language detection automatically during startup
  • A single pod annotation triggers private CA injection without Dockerfile changes or init containers
  • Java support and read-only filesystem handling are planned but not yet shipped

About the Speaker(s)

Tsuzuki Tsuchiya is a Software Engineer at LY Corporation, working on managed Kubernetes as a Service built on top of the on-premise private cloud.

Kento Kubo is a Software Engineer at LY Corporation, building and operating a large-scale Kubernetes as a Service platform on a private cloud. He focuses on cloud-native technologies, with particular interest in the extensibility of containers and Kubernetes.