The early October 2021 Facebook outage generated a predictable phenomenon – couch epidemiologists became experts in little-known Bridging the Gap Protocol (BGP), including its Introvert and Extrovert variants. Unfortunately, I also witnessed several unexpected trips to Mount Stupid by people who should have known better.

To set the record straight: everyone’s been there, and the more vocal you tend to be on social media (including mailing lists), the more probable it is that you’ll take a wrong turn and end there. What matters is how gracefully you descend and what you’ve learned on the way back.

When I started creating the How Networks Really Work series, I wondered whether our subscribers (mostly seasoned networking engineers) would find it useful. Turns out at least some of them do; this is what a long-time subscriber sent me:

How Networks Really Work is great, it’s like looking from a plane and seeing how all the roads are connected to each other. I know networking just enough to design and manage a corporate network, but there are many things I have learned, used and forgotten along the way.

So, getting a broad vision helps me remember why I chose something and maybe solve my bad choices. There are many things that I may never use, but with the movement of all things in the cloud it’s great to know, or at least understand, how things really work.

Two weeks ago I replied to a battle-scar reaction to 7-layer OSI model, this time I’ll address a much more nuanced view from Russ White. Please read his article first (as always, it’s well worth reading) and when you come back we’ll focus on this claim:

The OSI Model does not accurately describe networks.

Like with any tool in your toolbox, you can view the 7-layer OSI model in a number of ways. In the case of OSI model, it can be used:

I should have known better, but I got pulled into another stretched VLANs for disaster recovery tweetfest. Surprisingly, most of the tweets were along the lines of you really shouldn’t be doing that and that would never work well, but then I guess I was only exposed to a small curated bubble of common sense… until this gem appeared in my timeline:

Interestingly, that’s exactly how IP works:

One of my readers sent me this interesting question:

It begs the question in how far graduated students with a degree in computer science or applied IT infrastructure courses (on university or college level or equivalent) are actually aware of networking fundamentals. I work for a vendor independent networking firm and a lot of my new colleagues are college graduates. Positively, they are very well versed in automation, scripting and other programming skills, but I never asked them what actually happens when a packet traverses a network. I wonder what the result would be…

I can tell you what the result would be in my days: blank stares and confusion. I “enjoyed” a half-year course in computer networking that focused exclusively on history of networking and academic view of layering, and whatever I know about networking I learned after finishing my studies.

I was speaking with a participant of an SDN event in Zurich after the presentations, and he made an interesting comment: whenever he experienced serious troubleshooting problems in his career, it was due to lack of understanding of networking fundamentals.

Let me give you a few examples: Do you know how ARP works? What is proxy ARP? How does TCP offload work and why is it useful? What is an Ethernet collision and when would you see one? Why do we need MLD in IPv6 neighbor discovery?

TL&DR: With the recent changes to online BGP labs, you can also use Aruba CX, Cisco IOS, Cisco IOS XE, Cisco IOS XR, Dell OS10, Junos, or VyOS as external lab devices in most lab exercises (you could always use these devices for the routers you worked on). Previously, you could choose between Arista EOS and FRRouting, both of which are (obviously) still supported.

One of the goals of the Online BGP Labs project was to create an environment in which you could practice the BGP features you were interested in without spending an inordinate amount of time preparing the lab.

For example, if you want to figure out why BGP wedgies work the way they do, you need at least four additional autonomous systems, two of them acting as upstream ISPs for your customer router, and at least one of them implementing BGP policies using BGP communities.

Some netlab users want to accurately replicate their physical network’s topology in a virtual lab. Ignoring the obvious caveats for a moment, the first hiccup is usually the interface naming. All bets are off if you’re using anything but Ethernet in your actual network, but even if you did standardize on Ethernet, the container/VM interface names might not match the physical ones.

netlab provided a solution for a long time – you can specify interface ifindex when attaching a node to a link. For example, use the following topology to connect Ethernet3 on R1 to Ethernet6 on R2:

Geoff Huston published an article supposedly describing the challenge of securing NTP, but as is usually the case, he couldn’t skip the prior art going all the way back (almost) to the formation of Earth.

Before coming to the how do we secure NTP section, you’ll learn everything about the wobbly Earth rotation, the changes in the Earth’s angular speed, the impact of tides, the smearing of leap seconds, the differences between UT1 and UTC, why we use quasars to measure time, and everything there is to know about NTP. Have fun!

The never-ending “we will replace developers” (or networking engineers) pipe dream didn’t start with the latest bout of AI hype (or SDN). As Stephan Schwab explains in his Why We’ve Tried to Replace Developers Every Decade article, it started with COBOL, the magic high-level programming language that businesspeople would use to write their own programs.

At least some of us know how well that ended. I was also unfortunate to be there for the 5GL hype, the forms-driven programming hype, the “everyone will solve every problem out there with Excel macros” (it does work for networking inventory, doesn’t it?), and a few others. So please excuse me if I remain a bit skeptical about the latest fad, even though I find it (like all the previous ones) very useful when used conservatively in limited domains.

I work on a laptop that loves to power down when not used (the right thing to do), which often breaks the SSH session to my netlab server (not so good).

Reconnecting is trivial. Figuring out which lab I was working on and where it lives on the disk after a few hours? That’s the annoying part.

We solved most of that ages ago with the netlab status --all command. It shows all running labs¹ and their directories, so you can quickly jump back to where you were. However, even that gets tedious the 100th time you have to do it.

Most vendors “discovered” anycast gateways when they tried implementing routing between MAC-VRFs in an EVPN environment and hit all the usual tripwires (more about that later). A few exceptions (like Arista) supported them on VLAN segments for over a decade, and it was a no-brainer to extend that support to VXLAN segments.

Want to try out how that works? The Anycast Gateways on VXLAN Segments lab exercise is just what you need.

You can run the lab on your own netlab-enabled infrastructure (more details), but also within a free GitHub Codespace or even on your Apple-silicon Mac (installation, using Arista cEOS container, using VXLAN/EVPN labs).

Something didn’t feel right as I tried to check whether the IPv4 ECMP I observed in the latest version of Arista cEOS containers works with my MPLS/anycast scenario. The forwarding tables seemed OK, but I wasn’t getting MPLS labels in the ICMP replies (see RFC 4950 for details), even though I know Arista EOS can generate them.

I decided to go down that rabbit hole and built the simplest possible BGP-free core (the addition of BGP will become evident in a few seconds) to investigate PE/P-router behavior:

When I started the Online BGP Labs project in mid-2023, Cumulus Linux still seemed like a good platform to use. You could run devices as virtual machines (we were still supporting VirtualBox) or in containers (containerlab was improving with every release), and it looked more polished than bare-bones FRRouting.

Things only went downhill from there (from the perspective of offering a free and easy-to-use solution with a CLI resembling commonly-used devices):

In October 2023, I was talking about Internet routing security at the DEEP conference in Zadar, Croatia. After explaining the (obvious) challenges and the initiatives aimed at making Internet routing more secure (MANRS), I made my usual recommendation: vote with your wallet. However, if you’re a company in Croatia (or Slovenia, or a number of other countries), you’re stuck.

While ISPs in Croatia might be doing a great job, none of them is a MANRS participant¹, so we don’t know how good they are. The situation is not much better in Slovenia; the only ISPs claiming to serve Slovenia are Anexia (a cloud provider) and Go6 Institute, the small network operated by my good friend (and True Believer in IPv6 and MANRS) Jan Žorž. Moving further north, there are decent choices in Austria, and tons of options in Germany or Switzerland. I must be living in a truly thrifty part of Europe.