AI
NetObserv: What the eBPF Agent Collects
NetObserv is an open-source Kubernetes operator that installs eBPF agents on every node. Each agent reads packet headers in the kernel, extracts source and destination IPs and ports, then correlates those with Kubernetes resources, injecting pod name, namespace, and service information into each flow record. Additional features include DNS decoding, TCP latency, packet drop capture, NAT tracking, and cloud availability zone enrichment. A pull request open at talk time adds TLS and QUIC usage tracking.
Live Demo: Spotting a DNS Typo in the Flow Table
The live demo used a running cluster with the DNS feature enabled. Filtering flows by “DNS name is not empty” exposed a deliberate typo introduced into one pod’s DNS query. The overview page showed DNS error counts rising in the charts. The network health panel surfaced a pre-built alert for too many NXDOMAIN errors in the “mesh arena” application and linked directly to a runbook explaining the cause. From alert to root cause took under a minute of navigation.
Why Network Policy Debugging Needs CNI Integration
Real clusters accumulate hundreds or thousands of NetworkPolicy objects across namespaces. Cluster-level AdminNetworkPolicy objects add a second persona, the cluster admin, whose rules affect namespace workloads silently. Before this integration, a dropped packet simply disappeared from the observability view with no explanation. OVN-Kubernetes translates each NetworkPolicy into an ACL. When a packet matches an ACL and is dropped or allowed, nothing previously told the observability layer which policy caused it.
How the eBPF Hook and psample Bridge the Gap
The integration hooks the NetObserv eBPF agent into the OVS psample kernel function. Every time a packet matches an ACL condition, the hook fires. Instead of copying each event to user space immediately, the agent aggregates ACL IDs and packet metadata into kernel hashmaps. That batching keeps overhead low. In user space, the OVN-Kubernetes pod holds the ACL-to-NetworkPolicy mapping, so the ACL ID is translated back into a named Kubernetes policy object before the flow record is written.
Demo: Network Events Field in Practice
The pre-recorded demo used two namespaces, blue and red. A first failed curl showed a new “network events” field saying the packet was dropped by an AdminNetworkPolicy named “drop Google DNS” in the egress direction, a policy the namespace owner had not created. A second failure, after adding a NetworkPolicy, revealed a port typo: 8080 was specified instead of 80. A later iteration showed a policy with no namespace selector was allowing traffic to an unintended yellow namespace, caught only because NetObserv showed the allow reason alongside the destination.
Q&A
What is the flow retention policy and does NetObserv manage storage? NetObserv exports flows to Grafana Loki as structured logs (retention configured via the Loki operator), derives Prometheus metrics, and also supports OpenTelemetry, Kafka, and IPFIX exporters. Storage management is fully delegated. ▶ 27:22
Can the network-events feature be extended to other CNIs such as Cilium or Calico? No current plan exists from the team, but pull requests are welcome. The integration requires changes on both sides, inside the CNI and inside NetObserv, and Cilium already has its own solution. ▶ 28:31
Is OpenStack with OVN a feasible integration target? The Red Hat OpenStack team has already used NetObserv and was considering the integration. Because OpenStack uses OVN, adding the right psample entries to the OVN database is technically straightforward, though no timeline was committed. ▶ 30:46
Notable Quotes
it was dropped by admin network policy by the name drop Google DNS in the egress direction. So even though you didn’t create any network policies in your namespace, there was something created by the admin Nadia Pinaeva · ▶ 18:18
you can see that we actually typed the wrong port. It was 8080 and not 80. So I allowed not what I actually wanted to allow. Nadia Pinaeva · ▶ 20:23
sometimes connections are allowed but for the wrong reason which is another problem because network policy is there to provide security for your cluster but if everything works no one will double check even though you may be allowing more than you wanted to originally. Nadia Pinaeva · ▶ 25:34
Keep in mind we have been just playing with one network policy this whole time. There are usually kind of hundreds and thousands of those. Nadia Pinaeva · ▶ 24:48
About the Speakers
Joel Takvorian has been at Red Hat since 2016. He worked first on Hawkular, a time-series database, then on Kiali, the observability console for Istio. In 2021 he started a new team and project focused on network observability, NetObserv.
Nadia Pinaeva is a Senior Software Engineer at NVIDIA working on Kubernetes networking. She collaborates with SIG-network-policy to improve network security for Kubernetes clusters and contributes to the OVN-Kubernetes network plugin.