Building network automation solutions

The Dynamic Negotiation of BGP Capabilities blog post generated almost no comments, apart from the #facepalm realization that a certain network operating system resets IBGP sessions when the sole EBGP session goes down, but there were a few interesting comments on LinkedIn and Twitter.

While most engineers easily relate to the awkwardness of bringing down a BGP session to enable new functionality (Tearing down BGP session, as a solution reminds me rebooting a host, as a solution.), it’s not as easy as it looks. As Adam Chappell put it “Dynamic capability renegotiation does tend to sound a bit like changing the tyres while still moving. Very neat if you can pull it off but so much to go wrong…”

Here’s a fun fact network automation pundits don’t want to hear: if you’re working with replaceable device configurations (as we did for the past 20 years, at least those fortunate enough to buy Junos), you already meet the Infrastructure-as-Code requirements. Storing device configurations in a version control system and using reviews and merge requests to change them (aka GitOps) is just a cherry on the cake.

When I made a claim along these same lines a few weeks ago during the Network Automation Concepts webinar, Vladimir Troitskiy sent me an interesting question:

Setting up a network automation development environment is an interesting task:

• You have to install a half-dozen tools, each one with tons of dependencies;
• SSH libraries like paramiko have to installed manually;
• Ansible modules for individual network devices might need extra libraries;
• Parsing tools invoked with Ansible Jinja2 filters have to be installed separately;
• Add your pet peeve here ;)

Now imagine having to do that for a dozen networking engineers and software developers working on all sorts of semi-managed laptops. Containers seem to be one of the sane solutions¹.

In his latest blog post, Tom Hollingsworth compares network device disaggregations with cord cutting (replacing cable TV subscription with Netflix and friends), and comes to the inevitable conclusion:

The idea is that you gain freedom and cheaper software. The hope is that you can build an enterprise network for half of what it would normally cost. The reality is that you’re going to gain less functionality and spend more time integrating things together on your own instead of just putting in a turnkey solution.

To rephrase it, you’ll design a snowflake network with snowflake devices. Good job – just because it makes sense for the FAANG club (or LinkedIn), it doesn’t mean you should be doing it.

After the (in)famous October 2021 Facebook outage, Corey Quinn invited me for another Screaming in the Cloud chat, this time focusing on what went wrong (hint: it wasn’t DNS or BGP).

We also touched on VAX/VMS history, how early CCIE lab exams worked, how BGP started, why there are only 13 root name servers (not really), and the transition from networking being pure magic to becoming a commodity. Hope you’ll enjoy our chat as much as I did.

Dmytro Shypovalov sent me his views on the hardware differences between routers and switches. Enjoy!

So, a long time ago routers were L3 with CPU forwarding and switches were L2 with ASIC. Then they had invented TCAM and L3 switches, and since then ASICs have evolved to support more features (QoS, encapsulations etc) and store more routes, while CPU-based architectures have evolved to specialised NPU and parallel processing (e.g. Cisco QFX) to handle more traffic, while supporting all features of CPU forwarding.

Network automation CI/CD pipeline seems to be the next hot thing, with vendors and bloggers describing in detail how you could get it done. How realistic is that idea for an average environment that’s barely starting its automation journey?

TL&DR: it will take a long time to get there, and lack of tests is the first showstopper.

The multi-threaded routing daemons blog post generated numerous in-depth comments here and on LinkedIn. As always, thanks a million for keeping me honest and providing more details or additional perspectives. Here are some of the best bits.

Jeff Tantsura provided the first dose of reality:

All modern routing protocols implementations are multi-threaded, with a minimum separation of adjacency handling, route calculations and update generation. Note - writing multi-threaded code for complex tasks is a non trivial exercise (you could search for thread safety and similar artifacts and what happens when not implemented correctly). Moving to a multi-threaded code in early 2010s resulted in a multi-release (year) effort and 100s of related bugs all around.

Dr. Tony Przygienda added his hands-on experience (he’s been developing routing protocol software for ages):

The Anycast Works Just Fine with MPLS/LDP blog post generated so much interest that I decided to check a few similar things, including running BGP-based anycast over a BGP-free core, and using BGP Labeled Unicast (BGP-LU).

The Big Picture

We’ll use the same physical topology we used in the OSPF+MPLS anycast example: a leaf-and-spine fabric (admittedly with a single spine) with three anycast servers advertising 10.42.42.42/32 attached to two of the leafs:

If you find smart NICs interesting, you’ll like the latest blog post by James Hamilton explaining how AWS emulated Xen environment on Nitro hardware to keep old VM instances running on new hardware.

After a brief overview of the AI/ML hype, Javier Antich continued the AI and ML in Networking webinar with the basics of underlying technologies, starting with the machine learning fundamentals.

A month ago I explained how using a BGP route reflector in a large-enough non-symmetrical network could result in suboptimal routing (or loss of path diversity or multipathing). I also promised to explain how Advertisement of Multiple Paths in BGP functionality¹ solves that problem. Here we go…

I extended the original lab with another router to get a scenario where one route reflector (RR) client should use equal-cost paths to an external destination while another RR client should select a best path that is different from what the route reflector would select.

More than a decade ago (before SD-WAN was even a thing) I wrote an article describing how easy it is to route different applications onto different links (MPLS/VPN versus IPsec tunnels) using a distance vector routing protocol (preferably BGP, although even RIP would work).

You might find it interesting that it’s possible to solve tough problems with good network design instead of proprietary unicorn dust, so I salvaged the article from some dusty archive, cleaned it up, polished it, and published it on ipSpace.net.

I wanted to write a blog post explaining the intricacies of Advertisement of Multiple Paths in BGP, got into a yak-shaving exercise when discussing the need to exchange BGP capabilities to enable this feature, and decided to turn it into a separate prerequisite blog post. The optimal path selection with BGP AddPath post is coming in a few days.

The Problem

Whenever you want to use BGP for something else than simple IPv4 unicast routing the BGP neighbors must agree on what they are willing to do – be it multiprotocol extensions and individual additional address families, graceful restart, route refresh… (IANA has the complete BGP Capability Codes registry).

Stefano Sasso took my “Don’t complain, submit a PR” advice seriously and did a wonderful job adding support for Mikrotik RouterOS and VyOS to netsim-tools, increasing the number of supported platforms to twelve. His additions are available in release 1.0.2 which also includes:

• Automatic creation of groups based on BGP AS numbers
• Group-wide node attributes
• Experimental support for Python plugins

Interested? Start with tutorials and installation guide which includes lab building instructions.