Got this question from a networking engineer attending the Building Next-Generation Data Center online course:

Has anyone an advice on LACP fast rate? When and why should you use it instead of normal LACP?

Apart from forming link aggregation groups, you can use LACP to detect link- and node failures (more details). However:

Distributed systems are complicated. Add networking to the mix, and you get traumatic challenges like the CAP theorem and Byzantine fault tolerance. Most of those challenges are unknown to engineers who have to suffer through the vendor marketing presentations, making it hard to determine whether the latest shiny gizmo works outside of PowerPoint.

I started collecting articles describing distributed-system gotchas years ago, wrote numerous blog posts on the topic in the heydays of the SDN Will Save the World lemming run, and organized them into the Distributed Systems Resources page.

EIGRP routing updates have always contained the next hop field (similar to BGP updates), which was unused until Cisco IOS release 12.3 when the no ip next-hop-self eigrp AS-number interface configuration command was implemented.

EIGRP does not set the next hop field by default. An EIGRP router receiving a routing update thus assumes that the next hop of the received routes is the sending router. This behavior usually works well, but prevents site-to-site shortcuts to be established in DMVPN networks, and results in suboptimal routing in some route redistribution scenarios.

One of the most common causes of Internet routing leaks is an undereducated end-customer configuring EBGP sessions with two (or more) upstream ISPs.

Without basic-level BGP knowledge or further guidance from the service providers, the customer network engineer¹ might start a BGP routing process and configure two EBGP sessions, similar to the following industry-standard CLI² configuration:

Brandon Hitzel published a detailed document describing various Internet WAN edge designs. Definitely worth reading and bookmarking.

Another phenomenal detective story published on Cloudflare blog: Unbounded memory usage by TCP for receive buffers, and how we fixed it.

TL&DR: Moving TCP window every time you acknowledge a segment doesn’t work well with scaled window sizes.

The interesting takeaways:

After introducing the routing protocols and explaining the basics of link-state routing it was time for implementation considerations including:

• Collecting local endpoint reachability information
• Finding neighbors and exchanging the collected information (hint: a link-state topology database is just a distributed key-value store)
• Running the SPF algorithm (including partial SPF details) and installing the results

One of my readers sent me this (paraphrased) question:

What I have seen in my network are multicast packets with the IP source address set to 0.0.0.0 and source port set to 0. Is that considered acceptable? Could I use a multicast IP address as a source address?

TL&DR: **** NO!!!

It also seemed like a good question to test ChatGPT, and this time it did a pretty good job.

Years ago I’ve been involved in an interesting discussion focusing on NTP authentication and whether you can actually implement it reliably on Cisco IOS. What I got out of it (apart from a working example) was the feeling that NTP and it’s implementation in Cisco IOS was under-understood and under-documented, so I wrote an article about it. Of course the web version got lost in the mists of time but I keep my archives handy.

TL&DR: Installing an Ethernet NIC with two uplinks in a server is easy¹. Connecting those uplinks to two edge switches is common sense². Detecting physical link failure is trivial in Gigabit Ethernet world. Deciding between two independent uplinks or a link aggregation group is interesting. Detecting path failure and disabling the useless uplink that causes traffic blackholing is a living hell (more details in this Design Clinic question).

Want to know more? Let’s dive into the gory details.