Yak-Shaving netlab Podman Support

Long, long time ago, in a long-forgotten PR adding support for Cumulus VX Ignite runtime, an off-the-cuff remark was made saying, “and this is how you use netlab with Podman”. That remark was quietly sitting in the documentation for years until someone (A) tried to use Podman and (B) found enough time to report that it doesn’t work with netlab.

I don’t know what that tells us: either very few people use Podman, or nobody (apart from a notable exception) cares enough to spend a few minutes telling us stuff doesn’t work1.

Anyway, based on Joey Buiteweg’s wonderful research, I was able to put together a working solution pretty quickly, but it did involve a lot of yak shaving (more bovine details).

The original netlab implementation tried to do things the easy way (which rarely works) – when the default clab runtime is set to podman, add the podman runtime to all nodes. That might have worked at the time the off-the-cuff remark was made2; recent containerlab versions check the Docker socket (and thus fail miserably when Docker is not installed) unless you use the -r CLI argument (which didn’t fit nicely into how netlab starts the underlying orchestration system) or specify the runtime in CLAB_RUNTIME environment variable.

The environment variable seemed like the perfect solution, and it worked well on my test server. Alas, it failed to work on more recent versions of Ubuntu. More research by @joebb97 and @jbemmel turned up the culprit: someone figured out it would be great fun to break sudo. Gee, thanks a million, guys. That was really appreciated.

Back to the drawing board. Fortunately, new versions of containerlab support sudoless operation, but it requires the user to be in the clab_admins group. Removing sudo from start containerlab commands was easy, and a few warnings in the documentation should hopefully be enough3.

With that hurdle out of the way, we could pass the CLAB_RUNTIME variable to containerlab, and it would happily use Podman runtime (and stop complaining about the missing Docker socket). Adding a Podman installation script (you can do netlab install podman now) was a breeze after I figured out how to start the service that provides the socket-based API.

Next: quite a few docker commands are baked into the netlab source code4. Fortunately, Podman provides a Docker-like CLI that implements most of the docker commands we use. Dodged that bullet 🎉.

However, Podman has a different idea of who can see a running container than Docker. By default, only the user who starts the container can see it, and because containerlab has to start containers as root to create vEth pairs between them, a regular user cannot see them. End result: you could start the lab, but you couldn’t connect to the lab devices. For the moment, we ignored this inconvenience and documented that you MUST run Podman-based labs as root 🤷🏻‍♂️ (turning a bug into a fad5).

That should be it, right? I thought so, and the feature shipped in release 26.08. As is so often the case, the dirty details surfaced when the code met reality:

  • Podman provides a Docker-like CLI, but its JSON outputs don’t exactly match the corresponding Docker commands (gee, that’s REALLY useful).
  • Docker has a built-in DNS server that we used when creating the container’s /etc/resolv.conf files. Podman uses the host’s DNS resolver when creating those files, and the Docker-like settings we used to implement DNS clients on Linux-based containers failed to work with Podman. That’s been fixed as well.

These fixes are coming in the next netlab release (planned for early September 2026), or you could install the dev version.


  1. There’s the third option that I’m sure my lovely anonymous troll will eventually figure out: nobody is using netlab. However, I have approximately a dozen data points saying otherwise 🤷🏻‍♂️ ↩︎

  2. Although I doubt there was much more than idle speculation behind it, and yes, blame me for merging it 🤦‍♂️ ↩︎

  3. Most users shouldn’t be impacted. netlab checks containerlab version before starting the lab, and every containerlab upgrade should trigger the creation of the missing clab_admins group. ↩︎

  4. After programming for over 40 years, I still haven’t learned that every constant eventually becomes a variable. ↩︎

  5. Functions as Designed ↩︎

Add comment
Sidebar