Category: BGP

TL&DR: If you run multiple IGP protocols in your network, and add BGP on top of that, you might get the results you deserve. Even better, the results are platform-dependent.

One of my readers sent me a link to an interesting scenario described by Jeremy Filliben that results in totally unexpected behavior when using too many routing protocols in your network (no surprise there).

Imagine a network in which two edge routers advertise the same (external) BGP prefix. All other things being equal, it would make sense that other routers in the same autonomous system should use the better path out of the autonomous system. Welcome to the final tie-breaker in BGP route selection process: IGP metric.

We had the usual gloomy December weather during the end-of-year holidays, and together with the partial lockdown (with confusing ever-changing rules only someone in Balkans could dream up) it managed to put me in OCD mood… and so I decided to remove broken links from the old blog posts.

While doing that I figured out how fragile our industry is – I encountered a graveyard of ideas and products that would make Google proud. Some of those blog posts were removed, I left others intact because they still have some technical merits, and I made sure to write sarcastic update notices on product-focused ones. Consider those comments Easter eggs… now go and find them ;))

Continuing my archeological explorations, I found a dusty bag of old QoS content:

• Queuing Principles
• QoS Policing
• Traffic Shaping
• Impact of Transmit Ring Size (tx-ring-limit)
• FIFO Queuing
• Fair Queuing in Cisco IOS

I kept digging and turned out a few MPLS, BGP, and ADSL nuggets worth saving:

Remember my BGP route selection rules are a clear failure of intent-based networking paradigm blog post? I wrote it almost three years ago, so maybe you want to start by rereading it…

Making long story short: every large network is a unique snowflake, and every sufficiently convoluted network architect has unique ideas of how BGP route selection should work, resulting in all sorts of crazy extended BGP communities, dozens if not hundreds of nerd knobs, and 2000+ pages of BGP documentation for a recent network operating system (no, unfortunately I’m not joking).

More than a decade ago I published tons of materials on a web site that eventually disappeared into digital nirvana, leaving heaps of broken links on my blog. I decided to clean up those links, and managed to save some of the vanished content from the Internet Archive:

• OSPF Flooding Filters in Hub-and-Spoke Environments
• Implicit and Explicit Null Label in MPLS networks
• Default Routes in BGP
• Filter Excessively Prepended BGP Paths

I also updated dozens of blog posts while pretending to be Indiana Jones, including:

Did you ever experience an out-of-the-blue BGP session flap after you were running that peering for months? As Dmytro Shypovalov explains in his latest blog post, it’s always MTU (just kidding, of course it’s always DNS, but MTU blackholes nonetheless result in some crazy behavior).

Darren O’Connor put together a BGP looking glass with web GUI. Nothing fancy so far… but he also offers REST API interface (because REST API sounds so much better than HTTP).

The REST API calls return text results, so you can use them straight in a Bash script. For example, here’s a simple script to print a bunch of details about your current IP address:

The idea of redistributing the full Internet routing table (840.000 routes at this moment) into OSPF sounds as ridiculous as it is, but when fat fingers strike, it should be relatively easy to recover, right? Just turn off redistribution (assuming you can still log into the offending device) and move on.

Wrong. As Dmytro Shypovalov explained in an extensive blog post, you might have to restart all routers in your OSPF domain to recover.

And that, my friends, is why OSPF is a single failure domain, and why you should never run OSPF between your data center fabric and servers or VM appliances.

You might remember my occasional rants about lack of engineering in networking. A long while ago David Barroso nicely summarized the situation in a tweet responding to my BGP and Car Safety blog post:

If we were in a proper engineering we’d be discussing how to regulate and add safeties to an important tech that is unsafe and hard to operate. Instead, we blog about how to do crazy shit to it or how it’s a hot mess. Let’s be honest, if BGP was a car it’d be one pulled by horses.

The can we trust routing protocols series of blog posts I wrote in April 2020 (part 1, part 2, response from Jeff Tantsura) culminated in an interesting discussion with Russ White and Nick Russo now published as The Hedge Episode 43.

This podcast introduction was written by Nick Buraglio, the host of today’s podcast.

As we all know, BGP runs the networked world. It is a protocol that has existed and operated in the vast expanse of the internet in one form or another since early 1990s, and despite the fact that it has been extended, enhanced, twisted, and warped into performing a myriad of tasks that one would never have imagined in the silver era of internetworking, it has remained largely unchanged in its operational core.

The world as we know it would never exist without BGP, and because of the fact that it is such a widely deployed protocol with such a solid track record of “just working”, the transition to a better security model surrounding it has been extraordinarily slow to modernize.

I got this question about the use of AS numbers on data center leaf switches participating in an MLAG cluster:

In the Leaf-and-Spine Fabric Architectures you made the recommendation to have the same AS number on all members of an MLAG cluster and run iBGP between them. In the Autonomous Systems and AS Numbers article you discuss the option of having different AS number per leaf. Which one should I use… and do I still need the EBGP peering between the leaf pair?

As always, there’s a bit of a gap between theory and practice ;), but let’s start with a leaf-and-spine fabric diagram illustrating both concepts:

Got mentioned in this tweet a while ago:

Watching @ApstraInc youtube stream regarding BGP in the DC with @doyleassoc and @jtantsura.Maybe BGP is getting bigger and bigger traction from big enterprise data centers but I still see an IGP being used frequently. I am eager to have @ioshints opinion on that hot subject.

Maybe I’ve missed some breaking news, but assuming I haven’t my opinion on that subject hasn’t changed.

One of the attendees of our Building Next-Generation Data Center online course submitted a picture-perfect solution to scalable layer-2 fabric design challenge:

• VXLAN/EVPN based data center fabric;
• IGP within the fabric;
• EBGP with the WAN edge routers because they’re run by a totally different team and they want to have a policy enforcement point between the two;
• EVPN over IBGP within the fabric;
• EVPN over EBGP between the fabric and WAN edge routers.

The only seemingly weird decision he made: he decided to run the EVPN EBGP session between loopback interfaces of core switches (used as BGP route reflectors) and WAN edge routers.

Two weeks ago I started with a seemingly simple question:

If a BGP speaker R is advertising a prefix A with next hop N, how does the network know that N is actually alive and can be used to reach A?

… and answered it for the case of directly-connected BGP neighbors (TL&DR: Hope for the best).

Jeff Tantsura provided an EVPN perspective, starting with “the common non-arguable logic is reachability != functionality”.

Now let’s see what happens when we add route reflectors to the mix. Here’s a simple scenario: