Blog Posts in April 2023
Interesting: BGP Zombie Outbreak on Juniper Routers
BGP zombies are routes in the BGP table that refuse to disappear even though they should have been long gone. Recent measurements estimate between 0.5% and 1.5% of all routes in the global BGP table are zombies, which sounds crazy – after all, BGP is supposed to be pretty reliable.
Daryll Swer identified one potential source – Juniper routers do not revoke suppressed aggregated prefixes – and documented it in Navigating a BGP zombie outbreak on Juniper routers.
CloudFlare: From IP packets to HTTP
Want to know some details behind the CloudFlare SD-WAN implementation? You might find them in From IP packets to HTTP: the many faces of our Oxy framework.
I don’t know enough about Linux networking to figure out whether one could use those details to build something similar, but CloudFlare blog posts keep begin much better than Google’s Look How Awesome We Are recruitment drives.
Advantages and Drawbacks of EVPN-based Multihoming
Lukas Krattiger wrapped up his EVPN-versus-MLAG presentation (part of EVPN Deep Dive webinar) with an overview of the advantages and drawbacks of EVPN-based multihoming solutions:
- N-way multihoming
- Flexible connectivity (no need for a peer link)
- Fabric-wide scope (MAC multipathing required on ingress node)
Why Is OSPF (and BGP) More Complex than STP?
I got this question from one of my readers:
Why are OSPF and BGP are more complex than STP from a designer or administrator point of view? I tried everything to come to a conclusion but I couldn’t find a concluded answer, ChatGPT gave a circular loop answer.
There are numerous reasons why a protocol, a technology or a solution might be more complex than another seemingly similar one (or as Russ White would have said, “if you haven’t found the tradeoffs, you haven’t looked hard enough”):
Is Dynamic MAC Learning Better Than EVPN?
One of my readers worried about the control-plane-induced MAC learning lag in EVPN-based networks:
In all discussions about the advantages/disadvantages of VXLAN/EVPN, I can’t find any regarding the lag in learning new macs when you use the control plane for mac learning.
EVPN is definitely slower than data plane-based dynamic MAC learning (regardless of whether it’s done in hardware or software), but so is MLAG.
IPv6 Security in Layer-2 Firewalls
You can configure many firewalls to act as a router (layer-3 firewall) or as a switch bridge (layer-2 firewall). The oft-ignored detail: how does a layer-2 firewall handle ARP (or any layer-2 protocol)?
Unless you want to use static ARP tables it’s pretty obvious that a layer-2 firewall MUST propagate ARP. It would be ideal if the firewall would also enforce layer-2 security (ARP/DHCP inspection and IPv6 RA guard), but it looks like at least PAN-OS version 11.0 disagrees with that sentiment.
Straight from Layer 2 and Layer 3 Packets over a Virtual Wire:
Use Existing (DMVPN) Device Configurations in netlab
Anne Baretta decided to use netlab to test a proposed DMVPN topology. As netlab doesn’t support DMVPN (and probably never will), he decided to use netlab capabilities to start the lab topology and perform initial configuration, adding DMVPN configuration commands as custom configurations. Here’s how he described the process:
In this case I used netlab as a quick way to get a topology up and running, and then add the DMVPN configuration by hand.
Systems Design: What We Hope We Know
Avery Pennarun published a lovely rambling on magic, science, engineering and a pinch of AI. You might enjoy reading it1 with your Sunday morning coffee 😎.
180 Gbps Software-Only Linux Router
Pim van Pelt built an x86/Linux-based using Vector Packet Processor that can forwarding IP traffic at 150 Mpps/180 Gbps forwarding rates on a 2-CPU Dell server with E5-2660 (8 core) CPU.
He described the whole thing in a 8-part series of blog posts and a conference talk. Enjoy!
Video: SD-WAN CPE Architecture
Pradosh Mohapatra started the Typical SD-WAN Solution Architecture section of Software-Defined WAN (SD-WAN) Overview webinar with the backend architecture.
Next step: CPE architecture, the topic of today’s video.
Multi-Vendor EVPN Fabrics
Daniel left an interesting comment on my Studying EVPN to Prepare for a Job Interview blog post:
I also never build a VXLAN fabric with two vendors. So, is it possible now to build one large fabric consisting of multiple vendors?
TL&DR: Yes, but just because you could doesn’t mean that you should.
New: Network Infrastructure as Code Resources
While I was developing Network Automation Concepts webinar and the network automation online course, I wrote numerous blog posts on the Network Infrastructure as Code (NIaC) concepts, challenges, implementation details, tools, and sample solutions.
In March 2023 I collected these blog posts into a dedicated NIaC resources page that also includes links to webinars, sample network automation solutions, and relevant GitHub repositories.
IPv6 Addressing on Point-to-Point Links
One of my readers sent me this question:
In your observations on IPv6 assignments, what are common point-to-point IPv6 interfaces on routers? I know it always depends, but I’m hearing /64, /112, /126 and these opinions are causing some passionate debate.
(Checks the calendar) It’s 2023, IPv6 RFC has been published almost 25 years ago, and there are still people debating this stuff and confusing those who want to deploy IPv6? No wonder we’re not getting it deployed in enterprise networks ;)
Using VLAN and VRF Links in netlab Topologies
I already mentioned the introduction of VRF- and VLAN access links in netlab release 1.5.1. Let’s see how they can simplify your lab topologies.
I always tried to make lab topologies as concise as I could, sometimes cheating using JSON-in-YAML syntax. For example, the topology describing three routers running OSPF could be as simple as this:
module: [ ospf ]
nodes: [ r1, r2, r3 ]
links: [ r1-r2, r2-r3, r3-r1 ]
Let’s unravel that:
… updated on Sunday, April 16, 2023 15:50 UTC
ChatGPT Explaining the Need for iSCSI CRC
People keep telling me how well large language models like ChatGPT work for them, so now and then, I give it another try, most often resulting in another disappointment1. It might be that I suck at writing prompts2, or it could be that I have a knack for looking in the wrong places3.
This time4 I tried to “figure out5” why we need iSCSI checksums if we have iSCSI running over Ethernet which already has checksums. Enjoy the (ChatGPT) circular arguments and hallucinations with plenty of platitudes and no clear answer.
Worth Reading: Was MPLS TE Worth the Effort?
Bruce Davie continues documenting the tradeoffs we had to make in networking, this time with Was MPLS Traffic Engineering Worthwhile? I found this bit particularly familiar:
It wasn’t hard to make a theoretical argument that MPLS-TE could improve network performance and average link utilization, by moving traffic from congested links to uncongested ones. The hard part was proving that it would actually do a better job in practice than the more traditional methods such as using link weights and multipath routing to achieve the same ends.
Video: netlab Topology File
After introducing netlab in the Network Automation Tools webinar, I spent a few minutes describing the structure of the netlab lab topology file.
As always, use the video only as a starting point. For more details, read the netlab documentation (overview, reference guide).
Building a Small Network with ChatGPT
I must be a good prompt engineer – every time I ask ChatGPT something really simple it spews out nonsense. This time I asked it to build a small network with four routers:
I have a network with four Cisco routers (A,B,C,D). They are connected as follow: A-B, B-C, A-D, D-C. Each router has a loopback interface. Create router configurations that will result in A being able to reach loopback interfaces of all other routers.
Here’s what I got back1:
Here’s an example configuration for the four routers that should allow Router A to reach the loopback interfaces of all other routers:
Should I Care About RPKI and Internet Routing Security?
One of my subscribers sent me this question:
I’m being asked to enter a working group on RPKI and route origination. I’m doing research, listening to Jeff Tantsura, who seems optimistic about taking steps to improve BGP security vs Geoff Huston, who isn’t as optimistic. Should I recommend to the group that the application security is the better investment?
You need both. RPKI is slowly becoming the baseline of global routing hygiene (like washing hands, only virtual, and done once every blue moon when you get new IP address space or when the certificates expire). More and more Internet Service Providers (including many tier-1 providers) filter RPKI invalids thus preventing the worst cases of unintentional route leaks.
DHCP Relaying with Redundant DHCP Servers
Previous posts in this series (DHCP relaying principles, inter-VRFs relaying, relaying in VXLAN segments and relaying from EVPN VRF) used a single DHCP server. It’s time to add another layer of complexity: redundant DHCP servers.
Lab Topology
We’ll use a lab topology similar to the VXLAN DHCP relaying lab, add a second DHCP server, and a third switch connecting the two DHCP servers to the rest of the network.
Turning WiFi into a Thick Yellow Cable
The “beauty” (from an attacker perspective) of the original shared-media Ethernet was the ability to see all traffic sent to other hosts. While it’s trivial to steal someone else’s IPv4 address, the ability to see their traffic allowed you to hijack their TCP sessions without the victim being any wiser (apart from the obvious session timeout). Really smart attackers could go a step further, insert themselves into the forwarding path, and inject extra payload into unencrypted sessions.
A recently-discovered WiFi vulnerability brought us back to that wonderful world.
Kicking the Tires of GitHub Copilot
A friend sent me a video demo of his AI-driven network device configuration proof-of-concept. Before commenting on that idea, I wanted to see how well AI works as an assistant. Once Kristian Larsson mentioned he was using GitHub Copilot, it was obvious what to do next: try it out while working on the next netlab release.
TL&DR:
- It works.
- Some Copilot suggestions are uncannily accurate; others are fishing expeditions.
- It’s bland.
New: High Availability Clusters in Networking
Years ago I loved ranting about the stupidities of building stretched VLANs to run high-availability network services clusters with two nodes (be it firewalls, load balancers, or data center switches with centralized control plane) across multiple sites.
I collected pointers to those blog posts and other ipSpace.net HA cluster resources on the new High Availability Service Clusters page.
netlab Release 1.5.1: VLAN and VRF Links
netlab release 1.5.1 makes it easier to create topologies with lots of VRF- or VLAN access links, or topologies with numerous similar links. It also includes support for D2 diagram scripting language in case you prefer its diagrams over those generated by Graphviz.
Even if you don’t find those features interesting (more about them later), you might want to upgrade to fix a nasty container-related behavior I discovered in recently-upgraded Ubuntu servers.
DHCP Relaying in EVPN VRFs
After figuring out how DHCP relaying works and testing it with VRFs and in VXLAN segments, it seems like a no-brainer to make it work with EVPN.
TL&DR: It works, at least when using Arista vEOS as the relay and Cisco CSR 1000v as the DHCP server.
Lab Topology
We’ll keep using the exact same “physical” topology we used in the VXLAN DHCP relaying lab, add EVPN and BGP to the control-plane cocktail, and put the VXLAN segment into a VRF. We’ll use CSR 1000v as the DHCP server because Cisco IOSv doesn’t support some of the DHCP option-82 sub-options we need.
Worth Reading: The Death of CLI
Jeff McLaughlin wrote a nice blog post on the death of CLI (and why it has been greatly exaggerated):
The GUI-based layout tool [for iOS app development] is going away in favor of UI-as-code! The black screen always comes back!
As I’ve been saying for ages: people optimizing their productivity use CLI.