In this series of blog posts, we’ll explore numerous routing protocol designs that can be used to implement EVPN-with-VXLAN L2VPNs in a leaf-and-spine data center fabric. Every design will come with a companion netlab topology you can use to create a lab and explore the behavior of leaf- and spine switches.

Our leaf-and-spine fabric will have four leaves and two spines (but feel free to adjust the lab topology fabric parameters to build larger fabrics). The fabric will provide layer-2 connectivity to orange and blue VLANs. Two hosts will be connected to each VLAN to check end-to-end connectivity.

An RSS hiccup brought an old blog post from Urs Baumann into my RSS reader. I’m always telling networking engineers that it’s essential to set up realistic WAN environments when testing distributed software, and wemulate (a nice tc front-end) seemed like a perfect match. Even better, it runs in a container – an ideal component for a netlab-generated virtual WAN network.

wemulate acts as a bump in the wire; it uses Linux bridges to connect two container interfaces. We’ll use it to introduce jitter into an IP subnet:

┌──┐   ┌────────┐   ┌──┐
│h1├───┤wemulate├───┤h2│
└──┘   └────────┘   └──┘                       
◄──────────────────────►
     192.168.33.0/24    

Daryll Swer left a long comment describing how he designed a Service Provider network running in numerous private autonomous systems. While I might not agree with everything he wrote, it’s an interesting idea and conceptually pretty similar to what we did 25 years ago (IBGP without IGP, running across physical interfaces, with every router being a route-reflector client of every other router), or how some very large networks were using BGP confederations.

Just remember (as someone from Cisco TAC told me in those days) that “you might be the only one in the world doing it and might hit bugs no one has seen before.”

If you’re an Internet Service Provider running BGP with your customers, you might not want to send them the whole Internet routing table. Sending the regional prefixes and the default route is usually good enough.

Last week, I explained the differences between FRRouting and more traditional networking operating systems in scenarios where OSPF and IBGP advertise the same prefix:

• Traditional networking operating systems enter only the OSPF route into the IP routing table.
• FRRouting enters OSPF and IBGP routes into the IP routing table.
• On all platforms I’ve tested, only the OSPF route gets into the forwarding table¹.

One could conclude that it’s perfectly safe to advertise the same prefixes in OSPF and IBGP. The OSPF routes will be used within the autonomous system, and the IBGP routes will be propagated over EBGP to adjacent networks. Well, one would be surprised 🤦‍♂️

Now and then, someone rediscovers that IS-IS does not run on top of CLNP or IP and claims that, therefore, it must be a layer-2 protocol. Even vendors’ documentation is not immune.

Interestingly, most routing protocols span the whole seven layers of the OSI stack, with some layers implemented internally and others offloaded to other standardized protocols.

netlab release 1.7.0 added the fabric plugin that simplifies building lab topologies with leaf-and-spine fabrics. All you have to do to build a full-blown leaf-and-spine fabric is:

• Specify the default device type
• Enable the fabric plugin
• Specify the number of leaves and spines in the fabric.

For example, the following lab topology builds a fabric with Arista cEOS containers having two spines and four leaves:

Every time someone tries to persuade you to buy (expensive) big-buffer data center switches, take an antidote: the Things we (finally) know about network queues article by Avery Pennarun.

Last time, we discussed the first line of defense against fat finger incidents: limiting the number of BGP prefixes your router accepts from a BGP neighbor. However, you can do much more without deploying customer-specific filters (which might require a customer database) or ROV/RPKI.

You can practice the default filters you should always deploy on EBGP sessions with your customers in the Stop the Propagation of Configuration Errors lab exercise.

This is how we described the interactions between routing protocol tables, RIB, and FIB in the ancient times:

• Routing protocols compute the best paths to all known prefixes.
• These paths compete for entry in the routing table. The path(s) with the lowest administrative distance win.
• The entries from the routing table are fully evaluated (in particular, their next hops) and entered in the forwarding table.

Let’s use a simple BGP+OSPF network to illustrate what I’m talking about:

Arista EOS and Cisco Nexus OS got interface EBGP sessions years after Cumulus Linux. While they’re trivially easy to configure on FRRouting (the routing daemon used by Cumulus Linux), getting them to work on Arista EOS is a bit tricky.

To make matters worse, my Google-Fu failed me when I tried to find a decent step-by-step configuration guide; all I got was a 12-minute video full of YouTube ads. Let’s fix that.

I usually say that you cannot run netlab on Apple Silicon because the vendors don’t provide ARM images. However, when I saw an ARM version of the FRRouting container, I wondered whether I could run the BGP labs (admittedly only on FRR containers) on my M2 MacBook Pro.

TL&DR: Yes, you can do that.

Now for the recipe:

The March 2024 Internet Protocol Journal has a lengthy article on the history and “future” of Ethernet that might be worth reading (although it’s short on details) if you weren’t around when it all started.

Urs Baumann invited me to have a guest lecture in his network automation course, and so I had the privilege of being in lovely Rapperswil last week, talking about the basics of real-life network automation.

Urs published the video recording of the presentation on YouTube; hope you’ll like it, and if you don’t get too annoyed by the overly pushy ads, watch the other videos from his infrastructure-as-code course.

The “should we use the same vendor for fabric spines and leaves?” discussion triggered the expected counterexamples. Here’s one:

I actually have worked with a few orgs that mix vendors at both spine and leaf layer. Can’t take names but they run fairly large streaming services. To me it seems like a play to avoid vendor lock-in, drive price points down and be in front of supply chain issues.

As always, one has to keep two things in mind: