Why Packet-Level Knowledge Matters for Troubleshooting

▶ Watch (0:12)

Kubernetes networking hides its mechanics. You define a Service in YAML, pods run, traffic flows, and nothing breaks until something does. Without a mental model of the packet path, debugging becomes slow and painful. Rodigari built this talk around one goal: give engineers a traceable path from eth0 inside a pod all the way to a pod on a different node, so that when things break the path is already understood.

Kubernetes Networking Ground Rules and CNI Responsibilities

▶ Watch (3:41)

Kubernetes sets expectations but does not configure networking itself. Three ground rules apply: every pod gets a unique IP, no NAT occurs between pods communicating directly, and the network is flat at layer 3 so any pod can reach any other pod across nodes. When the scheduler places a pod, kubelet calls the container runtime, which creates a Linux network namespace, keeps it alive with a pause container, and then invokes the CNI plugin to configure the veth pair, the pod IP, and routes.

Veth Pairs, Overlay vs Underlay, and Cross-Node Traffic

▶ Watch (6:17)

Each pod connects to the host network namespace through a veth, a virtual cable with one end inside the pod namespace (eth0) and one end on the host. For cross-node traffic, CNIs choose between overlay and underlay. Overlay wraps the original pod-to-pod packet in an outer packet addressed to the destination node. VXLAN and IP-IP are common implementations. Underlay assumes the pod IP is routable on the physical network, sometimes called native routing, and uses BGP. Underlay skips the encapsulation overhead.

Services, DNAT, and How kube-proxy Rewrites Packets

▶ Watch (10:58)

Pod IPs change on restart, so direct pod-to-pod addressing is unreliable. A Kubernetes Service assigns a stable virtual cluster IP and load-balances across ready pods. No interface holds the cluster IP. Instead, kube-proxy in IP tables mode installs netfilter rules that perform DNAT: the destination cluster IP in the packet is rewritten to a backing pod IP before the packet leaves the node. eBPF-based CNIs do the same lookup via BPF maps with direct forwarding and no rule traversal, making them faster.

Live Demo: tcpdump Confirms DNAT and Load Balancing on the Host

▶ Watch (20:26)

The demo ran on a two-node kind cluster using kindnet and kube-proxy in IP tables mode. A debug pod on the same node as the front-end pod captured traffic with tcpdump using flags -i any -q -nn. When the front-end pod sent to the cluster IP 10.96.168.11, the first captured packet showed destination cluster IP. The second packet, leaving eth0, showed destination 10.244.2.15, the back-end pod IP. For kube-dns (10.96.0.10), IP tables held two entries (10.244.0.2 and 10.244.0.3) with statistic mode random at 50% each. A second DNS request hit the other CoreDNS pod, confirming load balancing.

Debugging Tools and Network Policy as a Next Step

▶ Watch (18:56)

Linux-native tools cover most debugging needs. The ip command inspects interfaces and routes. tcpdump captures packets. nslookup and dig validate DNS. kubectl debug creates an ephemeral container inside a running pod to inspect its network namespace, or creates a node-level pod to access the host network namespace. Rodigari also noted that Kubernetes allows any pod to reach any other pod by default. Network policy resources enforce per-pod firewall rules using IP tables or eBPF in the node data path.

Notable Quotes

Kubernetes networking can be invisible. Simone Rodigari · ▶ 00:44

simply means rewriting the destination IP while preserving the source IP. Simone Rodigari · ▶ 13:49

If we had 10 pods, we would see 10 vets. Simone Rodigari · ▶ 25:37

IP tables is using statistic mode random with a probability 50% for each of the two. Simone Rodigari · ▶ 31:23

the fact that Kubernetes by default allow any pod to communicate to any other pod, it’s not really sticking production. Simone Rodigari · ▶ 32:37

Key Takeaways

  • Each pod lives in its own Linux network namespace, connected to the host via a veth pair.
  • kube-proxy IP tables mode performs DNAT at the host, rewriting cluster IP to a pod IP before the packet exits.
  • kubectl debug gives live access to pod and host network namespaces without pre-installed tooling.

About the Speaker(s)

Simone Rodigari is a Software Engineer at Microsoft working on Azure Core Container Networking, focused on Kubernetes networking and observability. Previously certified in Kubernetes Administration and Security (CKA, CKS), they specialize in Go, eBPF, and cloud infrastructure.