Titus: A VM-Like Container Platform Built on Kubernetes

▶ Watch (1:04)

Titus is Netflix’s internal container platform. Users submit a container image with a resource request (say, four CPUs) and the system handles the rest. The platform migrated from virtual kubelet and Docker to kubelet and containerd with custom plugins. Workloads behave like VMs: containers run systemd, get SSH injected into their PID namespace, run as root inside a user namespace, and can bring their own storage. The compute runtime team owns the Kubernetes and Titus data plane and the base OS.

User Namespace Isolation and the inotify Accounting Bug

▶ Watch (5:15)

User namespaces assign each pod a unique UID range. Pod one might get UIDs 100,000-165,000, pod two 200,000-265,000. Set via pod.spec.hostUsers: false. The isolation failed for inotify: a pod used 500,000 inotify watches and exhausted systemd’s limit on the host. The kernel walks the user namespace tree up to root, so all pods shared containerd’s root-owned accounting bucket. A runC patch caps each container at 50,000 inotify watches. The full fix, landed in containerd 2.2.3, switches containerd to create the user namespace as the container user, so each pod owns its namespace and billing is fully isolated.

Per-Pod Network Isolation with a Custom CNI

▶ Watch (14:36)

Netflix wrote its own CNI called ITMÁN (IP Manager). Pods couldn’t use an off-the-shelf CNI because each pod needs its own security group and a harder network isolation model. Each pod gets a branch ENI attached to a trunk ENI on the host, with its own security group, subnet, and optionally its own VPC. Traffic is VLAN-tagged by branch ENI MAC address, so pods on the same host sit on different L2 domains. Bandwidth limits use TC with EDT (earliest departure time), with an eBPF program stamping each packet. A pod running iperf cannot flood the node.

Scalable Image Sharing with ID-Mapped Bind Mounts

▶ Watch (18:24)

containerd uses overlayFS for container root filesystems, with all pods sharing one backing disk. Netflix flips the “volatile” mount option to skip sync on ephemeral storage. For user namespaces, a pod running as user 100,000 cannot access root-owned image files. The old approach (chown every file per pod) destroyed overlay layer sharing. ID-mapped bind mounts fix this: a bind mount with an ID map attribute shifts UIDs by 100,000, giving the pod correct access without duplication. To cut mount overhead, Netflix bind-mounts the parent directory once instead of each layer, reducing the operation from O(m*n) to O(m).

Container Creation: Coordination, Profiles, and Resource Limits

▶ Watch (24:54)

Container creation at Netflix uses three extension points: NRI hooks, OCI runtime hooks, and a runC wrapper. The wrapper replaces the container entry point with “tiny” (init spelled backwards), an init process that listens on a socket, creating a post-create checkpoint before the user workload starts. Sidecar processes (including the service mesh) read container config during this window, then signal ready. Storage attaches directly into the mount namespace. Seccomp and AppArmor profiles compose dynamically from the declared workload type. Rlimits prevent fork bombs, CPU manager pins latency-sensitive pods to specific hyperthreads, and IO Max cgroups cap disk throughput per pod.

Notable Quotes

it’s not running as root but somehow it’s affecting systemd on the host which is running as root. Andrew Halaney · ▶ 8:47

your agent deciding to run iperf and drown your node is just not a thing that can happen here. Nick Rutigliano · ▶ 16:44

if you use podspec host users, you’re you’re no longer going to have this problem as of I think two two three in containerd. Andrew Halaney · ▶ 14:02

Key Takeaways

  • User namespace accounting for inotify walked to the host root user, leaking watch limits across all pods until containerd 2.2.3.
  • ITMÁN gives each pod its own branch ENI and security group, with eBPF-enforced bandwidth limits that no pod can exceed.
  • ID-mapped bind mounts let all pods share overlayFS layers while each pod sees files under its own UID range.
  • The runC wrapper injects “tiny” as the container init, blocking workload start until storage and sidecars are ready.
  • Seccomp and AppArmor profiles compose dynamically from the declared workload type, not set statically in the podspec.

About the Speaker(s)

Nick Rutigliano is a Senior SRE at Netflix on the Compute team. Previously he was a Staff Engineer at Spotify in Compute and Networking.

Andrew Halaney is a Senior Software Engineer at Netflix. His background spans Linux systems from automotive infotainment to the Kubernetes dataplane, with a focus on system software including bootloaders, kernel drivers, systemd, and container runtimes.