It looks like we’re bound to experience a widespread BGP failure once every few months. They all follow the same pattern:

• A “somewhat” undertested BGP implementation starts advertising paths with “unexpected” set of attributes.
• A specific downstream BGP implementation (and it could be a different implementation every time) a few hops down the road hiccups and sends a BGP notification message to its upstream neighbor.
• BGP session must be reset following a notification message; the routes advertised over it are lost and withdrawn, causing widespread ripples across the Internet.
• The offending session is reestablished seconds later and the same set of routes is sent again, causing the same failure and a session reset. If the session stays up long enough, some of the newly received routes might get propagated and will flap again when the session is reset.
• The cyclical behavior continues until a manual intervention.

Someone started an interesting discussion on the NANOG mailing list. He inherited a network that extended its internal OSPF to its multihomed customers and wondered whether he should leave the network, change OSPF to IS-IS, or deploy BGP. Here are a few thoughts from my reply.

You might think that the lack of a decent session layer in the TCP/IP protocol suite is the main culprit for our reliance on IP multihoming and related explosion of the IP routing tables. Unfortunately, we have an even bigger problem: the Berkeley Socket API, which is around 40 years old and used in almost all TCP/IP software implementations and clients (including high-level scripting languages like PERL or Python).

If you’ve ever tried to get advanced Cisco certifications, you’ve probably encountered questions dealing with the mismatch between the end device ARP timeouts and the L2 switch CAM (MAC address cache) timeouts. If you’re still wondering what the underlying problem is (it took me a while to figure it out), read the Unicast Flooding in Switched Campus Networks document from Cisco.

In all scenarios, traffic sent to unknown unicast MAC address causes layer-2 flooding, which can significantly reduce switch performance. Microsoft took this problem to a completely new level with its Network Load Balancing implementation: Windows servers send ARP replies containing MAC address X from MAC address Y, causing all the traffic toward the servers to be flooded – effectively turning an Ethernet switch into a hub.

I’ve sent a link to my Filter excessively prepended AS-paths article as an answer to a BGP route-map question to the NANOG mailing list and got several interesting questions from Dylan a few hours later. As they are pretty common, you might be interested in them as well.

In my environment, we are not doing full routes. We have partial routes from AS X and then fail to AS Y. Is their any advantage for someone like me to do this, as we are not providing any IP transit so we are not passing the route table to anyone else?

One of the biggest challenges facing the Internet core today is the explosion of the IP routing and forwarding tables, which is caused primarily by traffic engineering and multihoming requirements. Things were supposed to get better when IPv6 introduced strict hierarchical addressing (similar to the phone number addressing, where the first few digits always denote the country code).

Unfortunately, the hierarchical IPv6 addressing idea relied on incredible belief that the world will shape itself according to the wills of the IETF working group members. Not surprisingly, that didn’t happen and the hierarchical IPv6 addressing idea was quietly scrapped, giving us plenty more prefixes to play with when trying to pollute the global IPv6 routing tables.

Soon after I wrote the Designing Fast Converging BGP Networks article (you’ll find it somewhere in this list, one of my regular readers sent me an interesting problem: BGP sessions would be lost in his (IS-IS based) core network if he would use fall-over on IBGP neighbors and the BGP router would have a primary and a backup path to the IBGP neighbor.

It turned out to be an interesting side effect of aggressive route table purge following a link failure: the route to BGP neighbor was removed from the routing table before IS-IS ran SPF and installed an alternate route, and BGP decided it’s time to give up and terminate the session.

For more details, read the What Exactly Happens after a Link Failure blog post.

read more see 2 comments

Fifteen years ago, the focus of the “true” service provider was on voice traffic and data offerings based on virtual circuits, implemented with a plethora of semi-compatible technologies slowly developed within the ITU organization: X.25, ISDN, Frame Relay and the all-encompassing ATM.
In the meantime, some relatively small companies (including Cisco, Wellfleet and 3Com) were producing so-called “routers” that supported two technologies nobody took seriously: Ethernet and IP.

If you have a Kindle, you can subscribe to the Kindle edition of the Cisco IOS Hints blog. The monthly subscription charged by Amazon is $0.99 (the price is set by them and I cannot influence it) and (in the spirit of full disclosure :) I get 30% of the revenue.