Category: Cumulus Linux
I spent a rainy day implementing VLANs, VRFs, and VXLAN on Cumulus Linux VX and came to “appreciate” the beauties of Linux networking configuration.
TL&DR: It sucks
There are two major ways of configuring data plane constructs (interfaces, port channels, VLANs, VRFs) on Linux:
When I finally1 managed to get SR Linux running with netlab, I wanted to test how it interacts with Cumulus VX and FRR in an OSPF+BGP lab… and failed. Jeroen Van Bemmel quickly identified the culprit: MTU. Yeah, it’s always the MTU (or DNS, or BGP).
I never experienced a similar problem, so of course I had to identify the root cause:
The designers of Cumulus Linux CLI were always focused on simplifying network device configurations. One of the first features along these lines was BGP across unnumbered interfaces, then they introduced simplified EVPN configurations, and recently auto-MLAG and auto-BGP.
You can watch a short description of these features by Dinesh Dutt and Pete Lumbis in Simplify Network Configuration with Cumulus Linux and Smart Datacenter Defaults videos (part of Cumulus Linux section of Data Center Fabrics webinar).
After solving the BGP configuration challenge (could you imagine configuring BGP in a leaf-and-spine fabric with just a few commands in 2015), they did the same thing with EVPN configuration, where they decided to implement the simplest possible design (EBGP-only fabric running EBGP EVPN sessions on leaf-to-spine links), resulting in another round of configuration simplicity.
Linux operating system is used as the foundation for numerous network operating systems including Arista EOS and Cumulus Linux. It provides most networking constructs we grew familiar with including interfaces, VLANs, routing tables, VRFs and contexts, but they behave slightly differently from what we’re used to.
In Software Gone Wild Episode 86 Roopa Prabhu and David Ahern explained the fundamentals of packet forwarding on Linux, and the differences between Linux and more traditional network operating systems.
Most networking operating systems include a mechanism to roll back device configuration and/or create configuration snapshots. These mechanisms usually work only for the device configuration, but do not include operating system images or other components (example: crypto keys).
Now imagine using RFC 1925 rule 6a and changing the “configuration rollback” problem into “file system snapshot” problem. That’s exactly what Cumulus Linux does in its newest release. Does it make sense? It depends.
Mr. A. Anonymous left this comment on my BGP in the data centers blog post:
BGP is starting to penetrate into servers as well. What are your thoughts on having BGP running from the servers themselves?
Finally some people got it. Also, welcome back to the '90s (see also RFC 1925 section 2.11).
A few days ago Dell announced their next-generation network OS based on Debian Linux, and bloggers (like my good friend Tom Hollingsworth) started wondering what’s going to happen with Cumulus Linux.
Let’s get into prognostication mode…
Do you want to know more about Cumulus Linux after learning what data center architectures it supports, what base technologies it uses, and how you can use it to simplify network configurations? It’s time to explore Cumulus Linux architecture (part 5 of the presentation Dinesh Dutt had during the Data Center Fabrics webinar).
Dinesh Dutt started his part of the Data Center Fabrics Update webinar with “what is Cumulus Linux all about” and “what data center architectures does it support” and then quickly jumped into details about the base technologies used by Cumulus Linux: MLAG and IP routing.
Not surprisingly, the MLAG part generated tons of questions, and Dinesh answered all of them, even when he had to say “we don’t do that”.
After introducing the concepts of Cumulus Linux in the Data Center Fabrics update session, Dinesh Dutt described the typical data center architectures implemented with Cumulus Linux and the lessons everyone should learn from large-scale web properties.
With the advent of layer-3 leaf-and-spine data center fabrics, it became (almost) possible to build pure layer-3-only data center networks… if only the networking vendors would do the very last step and make every server-to-ToR interface a layer-3 interface. Cumulus decided to do just that.
As expected, he started with the big picture: what is Cumulus Networks and Cumulus Linux all about?
BGP is without doubt the most scalable routing protocol, which made it a popular choice for large-scale deployments from service provider networks to enterprise WAN/VPN networks and even data centers. Its only significant drawback is the tedious configuration process (which almost reminds me of writing COBOL programs decades ago).
A year ago Matthew Stone first heard about Cumulus Linux when I ranted about it on a Packet Pushers podcast (which only proves that any publicity is good publicity even though some people thought otherwise at that time), and when his cloud service provider company started selecting ToR switches he considered Cumulus together with Cisco and Arista… and chose Cumulus.