Blog Posts in January 2023
Design Clinic: Small-Site IPv6 Multihoming
I decided to stop caring about IPv6 when the protocol became old enough to buy its own beer (now even in US), but its second-system effects keep coming back to haunt us. Here’s a question I got for the February 2023 ipSpace.net Design Clinic:
How can we do IPv6 networking in a small/medium enterprise if we’re using multiple ISPs and don’t have our own IPv6 Provider Independent IPv6 allocation. I’ve brainstormed this with people far more knowledgeable than me on IPv6, and listened to IPv6 Buzz episodes discussing it, but I still can’t figure it out.
netlab Release 1.5.0: Larger Lab Topologies
netlab release 1.5.0 includes features that will help you start very large lab topologies (someone managed to run over 90 Mikrotik routers on a 24-core server):
- You can start libvirt virtual machines in batches to reduce the CPU overload that causes startup failures on large topologies.
- You can combine virtual machines and containers in the same lab, further reducing the memory footprint for devices available as true containers (Linux hosts, Cumulus/FRR routers, Arista cEOS)
- Use custom management network IP subnet if you’re running out of management IP addresses
To get more details and learn about additional features included in release 1.5.0, read the release notes. To upgrade, execute pip3 install --upgrade networklab.
New to netlab? Start with the Getting Started document and the installation guide.
Worth Reading: A Debugging Manifesto
Julia Evans published another fantastic must-read article: a debugging manifesto. Enjoy ;)
MUST READ: Nothing Works
Did you ever wonder why it’s impossible to find good service company, why most software sucks, or why networking vendors can get away with selling crap? If you did, and found no good answer (apart from Sturgeon’s Law), it’s time to read Why is it so hard to buy things that work well? by Dan Luu.
Totally off-topic: his web site uses almost no CSS and looks in my browser like a relic of 1980s. Suggestions how to fix that (in Chrome) are most welcome.
Built.fm Podcast with David Gee
I had a lovely chat with David Gee on his built.fm podcast sometime in December. David switched jobs in the meantime, and so it took him a bit longer than expected to publish it. Chatting with David is always fun; hope you’ll enjoy our chat as much as I did.
Hiding Malicious Packets Behind LLC SNAP Header
A random tweet1 pointed me to Vulnerability Note VU#855201 that documents four vulnerabilities exploiting a weird combination of LLC and VLAN headers can bypass layer-2 security on most network devices.
The security researcher who found the vulnerability also provided an excellent in-depth description focused on the way operating systems like Linux and Windows handle LLC-encapsulated IP packets. Here’s the CliffNotes version focused more on the hardware switches. Even though I tried to keep it simple, you might want to read the History of Ethernet Encapsulation before moving on.
Response: Network Automation Expert Beginners
I usually post links to my blog posts to LinkedIn, and often get extraordinary comments. Unfortunately, those comments usually get lost in the mists of social media fog after a few weeks, so I’m trying to save them by reposting them as blog posts (always with original author’s permission). Here’s a comment David Sun left on my Network Automation Expert Beginners blog post
The most successful automation I’ve seen comes from orgs who start with proper software requirements specifications and more importantly, the proper organizational/leadership backing to document and support said infrastructure automation tooling.
Will DPUs Change the Network?
It’s easy to get excited about what seems to be a new technology and conclude that it will forever change the way we do things. For example, I’ve seen claims that SmartNICs (also known as Data Processing Units – DPU) will forever change the network.
TL&DR: Of course they won’t.
Before we start discussing the details, it’s worth remembering what a DPU is: it’s another server with its own CPU, memory, and network interface card (NIC) that happens to have PCI hardware that emulates the host interface cards. It might also have dedicated FPGA or ASICs.
netlab: Building a Layer-2 Fabric
A friend of mine decided to use netlab to build a simple traditional data center fabric, and asked me a question along these lines:
How do I make all the ports be L2 by default i.e. not have IP address assigned to them?
Trying to answer his question way too late in the evening (I know, I shouldn’t be doing that), I focused on the “no IP addresses” part. To get there, you have to use the l2only pool or disable IPv4 prefixes in the built-in address pools, for example:
Worth Reading: Do We Need Network Automation
A long, long time ago, Mircea Ulinic (the author of Salt networking modules) wrote a long and thoughtful blog post on whether we need network automation (TL&DR spoiler: yes).
After reading the article, you might want to listen to the Salt and SaltStack podcast we did with Mircea a long while ago, and watch his presentation in Building Network Automation Solutions online course (also accessible with Expert Subscription).
Worth Exploring: OMNI and AERO
Do you ever feel like we don’t have enough overlay networking technologies? Don’t worry, there’s always another one, for example Overlay Multilink Network Interface (OMNI) with Asymmetric Extended Route Optimization (AERO) services. Want to know more? Fred Templin described it in a series of overview articles on APNIC blog.
Video: 2023 Network Engineer Path to Success
David Bombal kindly invited me to have another chat talking about the future of networking in late 2022. The resulting (masterfully edited) video is already on YouTube. Hope you’ll enjoy it as much as I enjoyed chatting with David.
Relationships between Layer-2 (VLAN) and Layer-3 (Subnet) Segments
Sometimes it takes me years to answer interesting questions, like the one I got in a tweet in 2021:
Do you have a good article describing the one-to-one relation of layer-2 and layer-3 networks? Why should every VLAN contain one single L3 segment?
There is no mandatory relationship between multi-access layer-2 networks and layer-3 segments, and secondary IP addresses (and subnets) were available in Cisco IOS in early 1990s. The rules-of-thumb1 claiming there should be a 1:1 relationship usually derive from the oft-forgotten underlying requirements. Let’s start with those.
Feedback: Docker Networking Deep Dive
While the pundits keeps telling me Docker is dead (looking at its documentation I would say they’re right) and Kubernetes it the way to go (yay!), some people still have to deal with Docker networking, and at least some of them found the Docker Networking Deep Dive webinar useful. Here’s a recent review:
You can scroll over internet pages as long as you can, you will rarely find this kind of specialized knowledge. This is the next level in term of knowledge about Docker.
If you belong to the “Kubernetes will rule the world” camp, we have you covered as well: Stuart Charlton created a phenomenal Kubernetes Networking Deep Dive webinar (approximately half of it is already accessible with free subscription).
Network Automation Expert Beginners
Some network automation skeptics came to that place the hard way: they got burned by half-baked semi-tested systems. This is what one of my good friends had to say in a LinkedIn comment:
I am suspicious of automation, as I’ve unfortunately seen too many outages caused by either human error or faulty automation. Every time it required human CLI/GUI intervention to correct it. The problem is that the more automation we push, the fewer people know how to use the “old school” way to administer stuff.
Network automation is not the only IT discipline that could cause hard-to-correct errors requiring manual intervention. I’m positive everyone knows at least one horror story resulting in manual tweaking of the Windows registry, or a sequence of arcane SQL commands1.
netlab Release 1.4.3: Cisco IOS XRv, MPLS on FRR
I had tons of plans to implement new netlab features during the last week of December, but then (fortunately) reality intervened and I spent my time relaxing and enjoying the break. I still managed to add IOS XRv support to netlab release 1.4.3 though ;). Other new features include:
- MPLS, LDP and L3VPN support on FRR by Oleg A. Arkhangelsky
- Optimized Linux container deployment that removes dependencies on Python and
ip - Custom templates for container configuration files
To upgrade, execute pip3 install --upgrade networklab.
New to netlab? Start with the Getting Started document and the installation guide.
Worth Reading: Routing Protocol Implementation Evaluation
In 2018 I tried to figure out whether the rush to deploy new routing protocols in leaf-and-spine fabrics is anything more than another blob of hype (RIFT, OpenFabric, BGP), considering OSPF got the job done for AWS. Those discussions probably sounded like a bunch of smart kids trying to measure outside temperature with a moist finger, so the only recommendation I could give in 2021 was “use the best tool for the job, keeping in mind you’re not Google or Microsoft”
It’s always better to measure than to have opinions, and a group of academics did just that. They developed Sybil – a tool to measure routing protocol performance in leaf-and-spine fabrics – and Dip Singh used it to compare BGP to IS-IS and OpenFabric.
Lack of Use Cases: a Lesson about Handling Hype
Gerben Wierda published an interesting article documenting how overhyped technologies eventually wither due to lack of realistic use cases.
He’s writing about blockchain, but it would be relatively trivial to replace that with OpenFlow – when was the last time you’ve seen something implemented with OpenFlow that wouldn’t be easier to do with traditional tools?
Video: MLAG with EVPN Deep Dive
In November 2022 I described some of the intricacies of using EVPN to implement MLAG control plane. You might have noticed that I didn’t dive deep into EVPN details, and I had a good reason for that – Lukas Krattiger did a wonderful job describing how MLAG works with EVPN in the EVPN Deep Dive webinar.
… updated on Tuesday, June 13, 2023 16:00 UTC
State of LDPv6 and 6PE
One of my readers successfully deployed LDPv6 in their production network:
We are using LDPv6 since we started using MPLS with IPv6 because I was used to OSPF/OSPFv3 in dual-stack deployments, and it simply worked.
Not everyone seems to be sharing his enthusiasm:
Now some consultants tell me that they know no-one else that is using LDPv6. According to them “everyone” is using 6PE and the future of LDPv6 is not certain.
Updates: AWS Networking
It didn’t make sense to update Amazon Web Services Networking webinar before the re:Invent conference – even though AWS introduced only a few networking features during the conference, at least one of them made a significant impact on the materials.
However, once the conference was over, I went over the to-do list that has been slowly accumulating for months and spent days updating over a dozen videos1. The major changes include:
Is It Time to Replace TCP in Data Centers?
One of my readers asked for my opinion about the provocative “It’s Time to Replace TCP in the Datacenter” article by prof. John Ousterhout. I started reading it, found too many things that didn’t make sense, and decided to ignore it as another attempt of a proverbial physicist solving hard problems in someone else’s field.
However, pointers to that article kept popping up, and I eventually realized it was a position paper in a long-term process that included conference talks, interviews and keynote speeches, so I decided to take another look at the technical details.
netlab Release 1.4.2: Juniper vMX and Junos Features
One of the last things I did before going on the Christmas break was to push out netlab release 1.4.2. Its highlights include:
- Juniper vMX by Stefano Sasso
- BFD, VRF, MPLS, SR-MPLS, and MPLS/VPN on Junos (also by Stefano)
- Full VLAN support on vMX and routed VLAN interfaces on vSRX (yet again, Stefano’s contribution)
- VyOS containerlab support by Oleg A. Arkhangelsky
- CSR 1000v VLAN and VXLAN support
Upgrading is as easy as ever: execute pip3 install --upgrade networklab.
New to netlab? Start with the Getting Started document and the installation guide.