Linux Networking Data Plane Configuration

I spent a rainy day implementing VLANs, VRFs, and VXLAN on Cumulus Linux VX and came to “appreciate” the beauties of Linux networking configuration.

TL&DR: It sucks

There are two major ways of configuring data plane constructs (interfaces, port channels, VLANs, VRFs) on Linux:

  • Use various CLI commands to add links, create bridges, make devices masters of other devices1 (that’s how you put a device into a port channel or VRF), or set up protocols like STP.
  • Use an intent-based system like ifupdown that takes a text configuration from /etc/network/interfaces, reads the current system state and adjusts the system state to be close to the administrator’s intent.

You don’t want to use the CLI commands to manipulate network devices; it’s a highway to CRUD hell and really hard to do right if you want to build idempotent configuration scripts (scripts that won’t crash and burn if you run them more than once). If you treasure your sanity, an intent-based system is the only way to go.

To make matters even more interesting, you can choose among a plethora of intent-based systems, all solving the same problem in slightly different ways. Focusing on Cumulus Linux made my life easier: it uses an improved version of ifupdown that knows how to handle the newer networking constructs like VRFs.

As expected from a Linux system utility, the documentation sucks. While the ifupdown documentation does a great job explaining the basics, once you try to configure bridges, VXLAN interfaces, or VRFs, you get into a maze of underdocumented commands, some mentioned only in the source code on GitHub. Fortunately, the Cumulus Linux documentation always describes multiple ways of configuring Linux networking objects, giving me at least some hints of what I should be searching for.

Next problem: netlab cannot build an aggregate configuration file. It uses a system of configurable modules (each implementing a technology or a protocol) and plugins, and every one of those could modify any part of the device configuration. For example:

  • The initial setup creates interfaces and assigns IP addresses to them
  • The VLAN module creates bridges and adds access and trunk VLANs to interfaces
  • The VRF module creates VRFs and makes them masters of other devices (bridges or interfaces)
  • The VXLAN module creates VXLAN tunnels and makes them part of the system bridge.

Each of the data plane netlab modules thus modifies objects already created or configured by other modules – a trivial exercise when dealing with a traditional network device that can add new configuration commands to an existing configuration. Doing that when trying to build a single text file quickly becomes a Mission Impossible.

Fortunately, ifupdown supports multiple configuration files and can even combine the desired state for a single interface from numerous entries spread across those files. It looked like a perfect fit until I hit another snag (this time, most probably related to my Ubuntu installation, as I couldn’t reproduce it on a freshly-built VM): trying to reload the state of all devices resulted in a weird error in the DHCP client.

Long story short, I couldn’t use ifreload -a to adjust the data plane state to the changed intent. I solved that with ifupdown classes (look at the netlab configuration scripts if you want to know the dirty details) and discovered all sorts of interesting quirks – I guess nobody ever tried to put a single device into a half-dozen classes.

But Wait, It Gets Worse

I thought I figured everything out, created templates using ifupdown for physical interfaces, bridges, VLAN interfaces, VLAN subinterfaces, VRFs, and VXLAN interfaces. Everything worked.

Next day, I decided to unify the Cumulus Linux and FRR container BGP configuration2, and found out that the unnumbered EBGP sessions wouldn’t start. Fortunately I had to deal with that in the past3 and knew exactly what the problem was: there was no IPv6 link-local address (LLA) on the interfaces (root cause: I was using an ancient Cumulus Linux container image, everything works with recent images).

As the unnumbered EBGP sessions worked in earlier netlab releases, I thought the failure was caused by my rewriting of the interface configuration template.

Hours later I came to the sad conclusion that:

  • It’s impossible to configure IPv6 LLA with ifupdown. Even iface inet6 auto (which would enable SLAAC, but also LLA) doesn’t work on Cumulus Linux.
  • Changing IPv6 interface parameters in FRR has no impact.
  • The only way to enable IPv6 on an interface is
    echo 0 >/proc/sys/net/ipv6/conf/_ifname_/disable_ipv6. WT**???

I hate using three mechanisms to configure a single object (an interface). If anyone knows a better way to enable IPv6 LLA (without a static IPv6 address) on a Linux interface, please write a comment.

Just in case this long rant confused someone: Cumulus Linux ships with IPv6 enabled (but not configured) on all interfaces, so you always get IPv6 link-local addresses, and unnumbered EBGP sessions work.

Now What?

To be fair, Cumulus engineers tried to solve the misery I just described with two systems: Network Command Line Utility (NCLU) in Cumulus Linux 4.x and NVIDIA User Experience (NVUE) in Cumulus Linux 5.x. Both of them look great until you hit their limits. The inability to enable IPv6 LLA4 on an interface without configuring a static IPv6 address is one of them5. No surprise there, they are both tweaking /etc/network/interfaces and FRR configuration. In the end, you’re forced to go back to the underlying configuration files and tweak system settings.

Not good enough? You could try Dell OS106 or VyOS – they both seem to be riding on top of Linux and use yet another syntax to configure the same networking objects using the same CLI commands or Netlink calls. As Andrew Tanenbaum said:

The nice thing about standards is that you have so many to choose from; furthermore, if you do not like any of them, you can just wait for next year’s model.

Some other network operating systems like Arista EOS or Cisco Nexus OS run on top of Linux, but they usually don’t use Linux networking objects but deal directly with ASICs.

  1. Everything is a device on Linux – physical interface, VLAN subinterface, port channel, bridge, VLAN interface, VRF, tunnel, loopback… You get the idea ;) ↩︎

  2. They are both using FRR, so it makes no sense to have two configuration templates doing the same stuff. ↩︎

  3. Read “I wasted three hours a few weeks ago, so I didn’t have to waste them now”. Cumulus Linux BGP Unnumbered documentation somehow fails to mention the “you need working IPv6 LLA” detail. ↩︎

  4. Something you could do on most major network operating systems for ages ↩︎

  5. But wait, there’s more. I have at least two other rants in the “to-write” queue. ↩︎

  6. Looking at the netlab initial configuration template Stefano Sasso created for Dell OS10, it has the ipv6 enable interface configuration command. ↩︎

Latest blog posts in CLI versus API series


  1. Dell OS10 "config plane" is based on Linux, but it's more like Arista... i.e., you can run bash & co, but it does not really use the standard Linux forwarding capabilities.

    On the other hand, VyOS it's really using the Linux standard forwarding plane, it's "only" a config layer for multiple backends (i.e., networking, FRR, Strongswan, ...).

    1. I use systemd’s networkctl for my configs and I think it’s okay. I managed to put together some unit templates that puts FRR and a set of interfaces into their own network namespace. I like it this way because no other services run in the same namespace unless I configure them to explicitly do so (with a other template). I put it up on GH so others can benefit:

    2. @Stefano: And this is how I learn something new every day. Thank you!

      @Jarad: Thanks a million!

    3. @Ivan on the "real world" devices, Dell OS10 works with Broadcom chipsets

  2. Just a reminder that NetPlan exists.

    It doesn't fix everything, but it will fix small issues like "I edited the static address in /etc/network/interfaces, restarted networking, and now I have an old and new address on that interface"

    I've used it in a few places, but not widely. And yes this does fall into the trap of "there are too many standards, I'm going to create one more to unify them all, except now it's just a +1"

  3. "The inability to enable IPv6 LLA4 on an interface without configuring a static IPv6 address is one of them."

    That's surprising to hear when you can configure BGP unnumbered using IPv6 LLA on Cumulus without specifying unicast IPv6 addressing on an interface.

    1. So the long rant did confuse someone. Straight from the blog post ;)

      > Cumulus Linux ships with IPv6 enabled (but not configured) on all interfaces, so you always get IPv6 link-local addresses, and unnumbered EBGP sessions work.

      In other words, you cannot enable or disable IPv6 packet processing on an interface with NCLU or NVUE (or at least I haven't found a way of doing that).

  4. Have you tried SONiC? Specific the Dell/Broadcom distribution?

    It comes with "sonic-cli" that make all of this a breeze

  5. Fortunately, on OpenBSD we have NSH that specifically addresses this problem: almost all system networking config handled from one CLI & stored in one file :-)

Add comment