Category: Cumulus Linux

A few weeks ago I described how Cumulus Linux tried to put lipstick on a pig reduce the Linux data plane configuration pains with Network Command Line Utility. NCLU is a thin shim that takes CLI arguments, translates them into FRR or ifupdown configuration syntax, and updates the configuration files (similar to what Ansible is doing with something_config modules).

Obviously that wasn’t good enough. Cumulus Linux 4.4 introduced NVIDIA User Experience¹ – a full-blown configuration engine with its own data model and REST API².

While ranting about Linux data plane configuration, I mentioned an interesting solution: Cumulus Linux Network Command Line Utility (NCLU), an attempt to make Linux networking more palatable to more traditional networking engineers.

NCLU is a simple wrapper around ifupdown2 and frr packages. You can execute net add and net del commands to set or remove configuration parameters¹, and NCLU translates those commands into changes to corresponding configuration files.

I spent a rainy day implementing VLANs, VRFs, and VXLAN on Cumulus Linux VX and came to “appreciate” the beauties of Linux networking configuration.

TL&DR: It sucks

There are two major ways of configuring data plane constructs (interfaces, port channels, VLANs, VRFs) on Linux:

When I finally¹ managed to get SR Linux running with netlab, I wanted to test how it interacts with Cumulus VX and FRR in an OSPF+BGP lab… and failed. Jeroen Van Bemmel quickly identified the culprit: MTU. Yeah, it’s always the MTU (or DNS, or BGP).

I never experienced a similar problem, so of course I had to identify the root cause:

The designers of Cumulus Linux CLI were always focused on simplifying network device configurations. One of the first features along these lines was BGP across unnumbered interfaces, then they introduced simplified EVPN configurations, and recently auto-MLAG and auto-BGP.

You can watch a short description of these features by Dinesh Dutt and Pete Lumbis in Simplify Network Configuration with Cumulus Linux and Smart Datacenter Defaults videos (part of Cumulus Linux section of Data Center Fabrics webinar).

When I first met Cumulus Networks engineers (during NFD9) their focus on simplifying switch configurations totally delighted me (video).

After solving the BGP configuration challenge (could you imagine configuring BGP in a leaf-and-spine fabric with just a few commands in 2015), they did the same thing with EVPN configuration, where they decided to implement the simplest possible design (EBGP-only fabric running EBGP EVPN sessions on leaf-to-spine links), resulting in another round of configuration simplicity.

Linux operating system is used as the foundation for numerous network operating systems including Arista EOS and Cumulus Linux. It provides most networking constructs we grew familiar with including interfaces, VLANs, routing tables, VRFs and contexts, but they behave slightly differently from what we’re used to.

In Software Gone Wild Episode 86 Roopa Prabhu and David Ahern explained the fundamentals of packet forwarding on Linux, and the differences between Linux and more traditional network operating systems.

Most networking operating systems include a mechanism to roll back device configuration and/or create configuration snapshots. These mechanisms usually work only for the device configuration, but do not include operating system images or other components (example: crypto keys).

Now imagine using RFC 1925 rule 6a and changing the “configuration rollback” problem into “file system snapshot” problem. That’s exactly what Cumulus Linux does in its newest release. Does it make sense? It depends.

Mr. A. Anonymous left this comment on my BGP in the data centers blog post:

BGP is starting to penetrate into servers as well. What are your thoughts on having BGP running from the servers themselves?

Finally some people got it. Also, welcome back to the '90s (see also RFC 1925 section 2.11).

A few days ago Dell announced their next-generation network OS based on Debian Linux, and bloggers (like my good friend Tom Hollingsworth) started wondering what’s going to happen with Cumulus Linux.

Let’s get into prognostication mode…

Do you want to know more about Cumulus Linux after learning what data center architectures it supports, what base technologies it uses, and how you can use it to simplify network configurations? It’s time to explore Cumulus Linux architecture (part 5 of the presentation Dinesh Dutt had during the Data Center Fabrics webinar).

Dinesh Dutt started his part of the Data Center Fabrics Update webinar with “what is Cumulus Linux all about” and “what data center architectures does it support” and then quickly jumped into details about the base technologies used by Cumulus Linux: MLAG and IP routing.

Not surprisingly, the MLAG part generated tons of questions, and Dinesh answered all of them, even when he had to say “We don’t do that.”

After introducing the concepts of Cumulus Linux in the Data Center Fabrics update session, Dinesh Dutt described the typical data center architectures implemented with Cumulus Linux and the lessons everyone should learn from large-scale web properties.

With the advent of layer-3 leaf-and-spine data center fabrics, it became (almost) possible to build pure layer-3-only data center networks… if only the networking vendors would do the very last step and make every server-to-ToR interface a layer-3 interface. Cumulus decided to do just that.

In May 2015 I invited Dinesh Dutt to talk about Cumulus Linux and its typical use cases on an update session of the Data Center Fabrics Architecture webinar.

As expected, he started with the big picture: what are Cumulus Networks and Cumulus Linux all about?