AI
Why CNI Stopped Working for Podman
Podman started on CNI in 2017 because it shared Kubernetes network semantics. But Podman only ever used 5% of CNI’s capabilities. Multi-node switching and complex plugins were irrelevant for a single-node engine. CNI also covered only 50% of what Podman needed: DNS solutions had to be written internally, and the Docker-compatible CLI clashed with Kubernetes-compatible networking output. In 2021, a CNI maintainer warned they might drop support for plugins Podman depended on. That forced a decision.
The Case for Rust Over Go
Three factors drove the Podman team to Rust over a familiar Go stack they’d used for four years. Aardvark-dns would run as a persistent daemon. For embedded deployments where idle resource usage matters, a garbage-collected language was a risk. In 2021, Go lacked generics. Error handling required custom wrapping libraries that felt awkward. Rust’s Result type with automatic error coercion and the question mark operator solved both problems at once.
Four Years of Netavark: What the Team Learned
Most Podman developers like Rust four years on, though the feeling isn’t unanimous. LLMs have changed the calculation. Developers who disliked Rust report they don’t mind it with a coding assistant. Code review is harder than Go: error patterns are easy to spot in Go at a glance, Rust’s advanced features make individual blocks harder to parse. The team also struggled to build internal Rust expertise. Netavark’s feature surface stayed small after the initial release, leaving few opportunities for deep skill-building. Conmon in Rust is next.
A UDP Bug With No Library to Fix It
When UDP packets arrive at a container before it starts listening, Linux’s conntrack caches a failure. The sender gets locked out until the entry times out. The fix was to flush stale conntrack entries the moment a container comes online. But Netavark runs in user space in Rust. Conntrack lives in the kernel. Communicating via netlink required a Rust library. One did not exist. Before Shivang could fix the bug in Netavark, he had to build the conntrack netlink library for the Rust community from scratch.
Building the Library, Fixing the Bug
Using netlink-packet-core crates, Shivang built Rust structs and serialization code for conntrack get, delete, and new commands. He submitted them upstream. Maintainers reviewed and merged them. With the library live, he hooked it into Netavark’s port forwarding setup. The fix matched the container’s IP against the conntrack reply tuple and the host port against the original tuple. On container start, Netavark sends a netlink delete message. Stale entries flush immediately. Integration tests turned green. The library is now available to anyone writing netlink code in Rust.
Q&A
Four years on with Go now having generics, would you make the same Rust call? Heon said the thought experiment mostly makes him wish Podman itself was in Rust, not the other way around. Developer availability is imperfect but LLMs reduce the barrier for getting contributors productive in a new language. ▶ 20:49
What were the biggest friction points moving a Go team to Rust? The team wrote a lot of bad Rust early on and is progressively refactoring. Taking the jump without experienced Rust mentors left the codebase in a suboptimal state. Compiler warnings help, but won’t tell you when to apply advanced features. ▶ 22:37
Is Netavark meant to replace slirp4netns? No. slirp4netns sits beneath Netavark for rootless containers and is being replaced by “pasta” at the host layer, not by Netavark. Podman 6 ships the migration automatically, coming the second week of May. ▶ 24:16
Notable Quotes
Chef’s kiss. We loved it. Matt Heon · ▶ 09:22
Key Takeaways
- Podman used only 5% of CNI and rewrote the networking layer in Rust when upstream support wavered
- aardvark-dns running as a persistent daemon made garbage collection a real concern, not a preference
- LLMs are narrowing the language-learning barrier but Rust code review remains harder than Go
- A missing Rust library for netlink/conntrack had to be written before the UDP bug could be fixed
- The new conntrack netlink library is now public and available to all Rust projects
About the Speaker(s)
Matt Heon is Principal Software Engineer at Red Hat and one of the core maintainers of Podman, Buildah, and Skopeo. He has led the Podman team since the project’s inception in 2017.
Shivang K Raghuvanshi is a computer science undergraduate and former LFX mentee with the Podman Container Tools project. As part of the LFX program, he built the first native Rust library for the Linux kernel’s Conntrack subsystem to fix a critical networking bug in Netavark.