While according to the GIFEE True Believers™, Docker is dead and Kubernetes rules the world, people who want to have a bit of life might be perfectly happy running “obsolete” stuff like Docker on their laptops or Linux VMs.

If you happen to be one of the latter, you might like the Introduction to Docker webinar I put together a few years ago. It’s now public; you can watch it with an ipSpace.net account.

Looking for more binge-watching materials? You’ll find them here.

I was not exactly surprised when it did not work. While Arista accepted FRRouting EVPN routes, the FRRouting BGP daemon rejected routes sent by Arista EOS:

On a tangential note, calling those blunders hallucinations was a marketing masterstroke. Not being a native English speaker, I might be missing some nuances, but I feel like hallucinations might be something you’re not responsible for (some of the time), whereas we all know who’s responsible for bullshit and shameless guesses – and responsibility is something the AI companies are clearly trying to stay as far away from as possible.

On another tangential note, if you’re not following Scott Alexander’s blog substack, you’re missing out.

However, this year Miguel Redondo fixed the netlab SR Linux configuration templates (VRF export policies, LocPref routing policy changes) before I could even start looking at them, and Roman Dodin released a tool that tells you exactly what changed between software releases and how to fix it.

AI is no exception, but of course, the AI evangelists, LinkedIn AI Wranglers¹, and Thought Leaders will never tell you that (or even admit it).

One of the goals of the Online BGP Labs project was to create an environment in which you could practice the BGP features you were interested in without spending an inordinate amount of time preparing the lab.

For example, if you want to figure out why BGP wedgies work the way they do, you need at least four additional autonomous systems, two of them acting as upstream ISPs for your customer router, and at least one of them implementing BGP policies using BGP communities.

netlab provided a solution for a long time – you can specify interface ifindex when attaching a node to a link. For example, use the following topology to connect Ethernet3 on R1 to Ethernet6 on R2:

Before coming to the how do we secure NTP section, you’ll learn everything about the wobbly Earth rotation, the changes in the Earth’s angular speed, the impact of tides, the smearing of leap seconds, the differences between UT1 and UTC, why we use quasars to measure time, and everything there is to know about NTP. Have fun!

At least some of us know how well that ended. I was also unfortunate to be there for the 5GL hype, the forms-driven programming hype, the “everyone will solve every problem out there with Excel macros” (it does work for networking inventory, doesn’t it?), and a few others. So please excuse me if I remain a bit skeptical about the latest fad, even though I find it (like all the previous ones) very useful when used conservatively in limited domains.

Reconnecting is trivial. Figuring out which lab I was working on and where it lives on the disk after a few hours? That’s the annoying part.

We solved most of that ages ago with the netlab status --all command. It shows all running labs¹ and their directories, so you can quickly jump back to where you were. However, even that gets tedious the 100th time you have to do it.

Want to try out how that works? The Anycast Gateways on VXLAN Segments lab exercise is just what you need.

You can run the lab on your own netlab-enabled infrastructure (more details), but also within a free GitHub Codespace or even on your Apple-silicon Mac (installation, using Arista cEOS container, using VXLAN/EVPN labs).

I decided to go down that rabbit hole and built the simplest possible BGP-free core (the addition of BGP will become evident in a few seconds) to investigate PE/P-router behavior:

Things only went downhill from there (from the perspective of offering a free and easy-to-use solution with a CLI resembling commonly-used devices):

While ISPs in Croatia might be doing a great job, none of them is a MANRS participant¹, so we don’t know how good they are. The situation is not much better in Slovenia; the only ISPs claiming to serve Slovenia are Anexia (a cloud provider) and Go6 Institute, the small network operated by my good friend (and True Believer in IPv6 and MANRS) Jan Žorž. Moving further north, there are decent choices in Austria, and tons of options in Germany or Switzerland. I must be living in a truly thrifty part of Europe.

nodes: [ r1, r2, h1, h2 ]

links:
- r1-r2
- interfaces: [ r1, r2, h1, h2 ]

This is what GraphViz generates based on netlab’s description of the lab topology:

While some of that screen scraping is pure inertia, we sometimes have good reasons to do it rather than use protocols like NETCONF, gNMI, or protobufs. In Episode 205 of Software Gone Wild, I’m discussing some of those reasons and exploring the gap between vendor theory and reality with Dinesh Dutt, who is unlucky enough to have become the world’s foremost expert on crappy network telemetry.

Arista cEOS got better in the meantime; IPv4 ECMP works like a charm on cEOS release 4.35.02F. With the same lab topology I’d used in 2021, I was able to see the traffic spread across multiple nodes:

I managed to avoid VPN clients most of my life, so I have no idea whether this is a “finally someone figured that out 🎉” moment or a late catch-up to what other VPN clients have been doing for ages. Feedback (in comments or otherwise) would be most welcome!

• New Cisco IOS XR features: SR-MPLS, MPLS/VPN, MPLS 6PE, EVPN over MPLS, VRRP, BGP session parameters, routing policies, prefix filters, AS-path filters, community filters, and static routes
• Juniper cRPD support (mostly by @leec-666)
• EVPN/VXLAN on Juniper vPTX and vJunos-router by @rickycraft
• OpenBSD BGP support by @remilocherer
• SR-MPLS support for OSPFv2 on EOS, FRR, IOS XE, IOS XR
• EVPN/MPLS improvements, including using SR-MPLS transport

For even more details, check the release notes.

That’s exactly what Sebastien d’Argoeuves did: he developed a solution that automates Cisco SD-WAN deployment after the corresponding netlab lab is started, and published it in a GitHub repo. If you’re an SD-WAN fan, you must give it a try ;)

• One of the PE-devices is the device we want to test
• The other PE-device is a device that is known to work (ideally, an FRRouting container).
• Bonus points if the other PE-device can generate operational data in JSON format. Using a device for which we already have a validation plugin is close to perfection.
• Add a P-router in the middle because MPLS.
• Attach some hosts to the two PE-devices (we’re testing two MAC-VRFs in the final version of the test)
• After validating everything that can reasonably be validated (OSPF session, IBGP session, EVPN AF on IBGP session), do the end-to-end pings and hope for the best.

This is the graph netlab created from the lab topology:

Other interesting data points from the article (and linked presentations):

• People are running (a few) thousands of routers in a single area
• Running Dijkstra’s algorithm on an emulated network with 2000 nodes took 100 msec… in 2003 (page 18 of this NANOG presentation).

It turns out (as I expected) that all the noise about the need for new routing protocols we were experiencing a few years ago was either due to bad implementations or coming from nerds looking for new toys to play with.

Here’s the lab topology we’ll use (also available on GitHub):

You can run the lab on your own netlab-enabled infrastructure (more details), but also within a free GitHub Codespace or even on your Apple-silicon Mac (installation, using Arista cEOS container, using VXLAN/EVPN labs).

Most platforms failed those tests because we forgot to configure route-reflector-clients in labeled IPv6 and VPNv4/VPNv6 address families¹. That was easy to fix, but the IOS-based devices were still failing the tests, with nothing in the toolchain ever complaining about configuration problems.

There was this post about AI agents speaking BGP with an associated GitHub repo, so I could go take a look at what it’s all about.

The proof-of-concept (so the post author) has two components:

I’m not sure I’m brave (or young) enough to try it out, but if you’re planning to organize a small gathering (like a local Network Operator Group), this might be an interesting, slightly more structured approach than a Net::Beer event. It would also be nice to know whether someone managed to pull it off in an online format.

If the use case is the initial lab deployment, why don’t you use containerlab startup-config option to change the device’s startup configuration?

I have to admit, I’m too old to boldly go with the just use the startup configuration approach. In ancient times, Cisco IOS did crazy stuff if you rearranged the commands in the startup configuration. But ignoring that historical trivia (Cisco IOS/XE seems to be doing just fine), there are several reasons why I decided to use the startup configurations (and you can use them with some containers) as the last resort:

While we’re mostly discussing EVPN in conjunction with VXLAN encapsulation, its initial use case was as an alternate control plane for MPLS networks.

Krzysztof Szarkowicz had a great presentation describing the specifics of EVPN in MPLS-Based Environments a few years ago. Those videos (part of the EVPN Technical Deep Dive webinar) are now public; you can watch them without an ipSpace.net account.

Looking for more binge-watching materials? You’ll find them here.

Use the passive command in appropriate mode to suppress the sending of OSPF protocol operation on an interface.

However, when I ran the OSPFv2 passive interface integration test with an IOS/XR container, it kept failing with neighbor is in Init state (the first and only time I ever encountered such an error after testing over two dozen platforms).

Starting with the disgusting configuration mechanism:

You don’t have to install VM or container managers (Vagrant/containerlab), or beg vendors to give you access to device VMs/containers, to get working device configurations. All you need is a Python package that works on Windows¹, macOS, or Linux.

It’s as simple as this:

If you already set up your own netlab environment, you probably know what to do (or you can get the details here). Alternatively, you can click here to start the lab in your browser using GitHub Codespaces. After starting the lab environment, change the directory to advanced/10-sr and execute netlab up.

Starting with the trivial hiccups:

• Support for Kubernetes (KinD) clusters based on work by @wnagele
• Layer-2 EVPN/VXLAN support on Cat8000v, IOL, and IOLL2
• netlab graph command can create graphs from a subset of nodes or links
• You can specify the parameters of core links in the fabric plugin
• OSPFv3 reports

The fun part, however, are the new container configuration methods:

containerlab and GNS3 are clear leaders (no surprise there) with the original vrnetlab becoming abandonware (fortunately, we have Roman Dodin’s fork), which makes me think we should focus on using netlab primarily with containerlab and slowly sunset the Vagrant support, particularly considering some people actively hate the license change.

Also, if anyone feels like writing an interface (provider module) between netlab and GNS3, the pull request would be most welcome 😎

Any thoughts? Please leave a comment!

There was just one tiny gotcha: all netlab-generated EOS configuration files are device configuration snippets that are intended to be submitted via EOS CLI, and I didn’t feel like cracking open the netmiko documentation (that’s another backburner project).

However, Arista cEOS includes this magic command called FastCli ;)

It’s time to move up the protocol stack. Let’s see how you can route between VXLAN segments, this time using unique unicast IP addresses on the layer-3 switches.

You can run the lab on your own netlab-enabled infrastructure (more details), but also within a free GitHub Codespace or even on your Apple-silicon Mac (installation, using Arista cEOS container, using VXLAN/EVPN labs).

The idea of speeding up the lookup operation on an IP datagram turned out to have little practical impact.

That might be true¹, although I do remember how hard it was for Cisco to build the first IP forwarding hardware in the AGS+ CBUS controller. Switching labels would be much faster (or at least cheaper), but the time it takes to do a forwarding table lookup was never the main consideration. It was all about the aggregate forwarding performance of core devices.

Anyhow, Duty Calls. It’s time for another archeology dig. Unfortunately, most of the primary sources irrecoverably went to /dev/null, and personal memories are never reliable; comments are most welcome.

Unfortunately, it seems that everything old is new again (see also RFC 1925 rules 4 and 11), as proved by a “Using GRE and VXLAN for Fun and Profit” (my summary) presentation at DEFCON 33. Even if you knew that unencrypted tunnels are insecure (duh!) for decades, you might still want to read the summary of the talk (published on APNIC blog) and view the slides.

Let me start with a bit of background:

• IOL Layer 2 image starts with interfaces enabled and in bridged (switchport) mode (details)
• netlab has to run a normalize script (applicable to IOLL2, IOSv L2, and Arista EOS) before configuring anything else to ensure all interfaces are shut down.
• The IOLL2 normalize Jinja template had a bug – when setting the interface MAC address, it checked l.mac_address instead of intf.mac_address. Nevertheless, everything worked because the MAC addresses were also set during the initial device configuration.

netlab uses two different mechanisms to configure FRR containers:

• Data-plane features are configured with bash scripts using ip commands and friends.
• Control-plane features are configured with FRR’s vtysh

I wanted to replace both with Linux scripts that could be started with the docker exec command.

The videos from the Network Observability webinar with Dinesh Dutt are now available without a valid ipSpace.net account. Enjoy!

I wanted to ask you how you think AI will affect networking jobs. What’s real and what’s hype?

Before going into the details, let’s make a few things clear:

So much opportunity, inspiration, creativity, and possibility lies in applying the skills and experience that you may have from technological disciplines in other realms and industries that are often far less advanced in their deployment of technologies.

As well as:

This too shall pass. One of the great gifts of working in technology is that it’s given so many of us the habit of constantly learning, of always being curious and paying attention to the new things worth discovering.

Hope you’ll find it helpful and at least a bit inspiring.

Now let’s add the EVPN control plane into the mix. The data plane (VLANs mapped into VXLAN-over-IPv4) will remain unchanged, but we’ll use EVPN (a BGP address family) to build the ingress replication lists and MAC-to-VTEP mappings.

For example, in the initial BGP lab, I didn’t want any BGP-related configuration on RTR while X1 would already be fully configured – when the student configures BGP on RTR, everything just works.

For Arista part - just use AVD with all templates included and ANTA for testing. I was always wondering why netlab is not doing that.

Like any other decent network automation platform, netlab uses a high-level data model (lab topology) to describe the network. That data model is then transformed into a device-level data model, and the device-level data structures are used to generate device configurations.

How is a GRE tunnel different compared to an MPLS LSP? I feel like conceptually, they kind of do the same thing. They just tunnel traffic by wrapping it with another header (one being IP/GRE, the other being MPLS).

Instead of going down the “how many angels are dancing on this pin” rabbit hole (also known as “Is MPLS tunneling?”), let’s focus on the fundamental differences between GRE/IPsec/VXLAN tunnels and MPLS paths.

Let’s start with the background story: due to the (now fixed) suboptimal behavior of bleeding-edge Ansible releases, I decided to generate the device configuration files within netlab (previously, netlab prepared the device data, and the configuration files were rendered in an Ansible playbook).

As we use bash scripts to configure Linux containers, it makes little sense (once the bash scripts are created) to use an Ansible playbook to execute docker exec script or ip netns container exec script. netlab release 26.01 runs the bash scripts to configure Linux, Bird, and dnsmasq containers directly within the netlab initial process.

Now for the juicy part.

I discussed these questions with Damien Garros, the driving force behind Infrahub, the founder of OpsMill (the company developing it), and a speaker in the ipSpace.net Network Automation course.

I fixed a few blog posts based on his feedback (I can’t tell you how much I appreciate receiving a detailed “you should fix this stuff” message, and how rare it is, so thanks a million!), and David was kind enough to add a delightful cherry on top of that cake with this wonderful blurb:

Netlab has been a lifesaver. Ivan’s entire approach, from the software to collecting instructions and providing a meaningful information trail, enabled me to go from zero to having a functional lab in minutes. It has been an absolute lifesaver.

I can be lazy with the infrastructure side, because he’s done all of the hard work. Now I get to concentrate on the value-added functionality of my own systems and test with the full power of an automated and modern network lab. Game-changing.

Here’s an interesting question I got from a reader in the midst of an OSPF-to-IS-IS migration:

Why should one bother with different [IS-IS] areas when the routing hierarchy is induced by the two levels and the appropriate IS-IS circuit types on the links between the routers?

Well, if you think you need a routing hierarchy, you’re bound to use IS-IS areas because that’s how the routing hierarchy is implemented in IS-IS. However…

Other new features include:

• EVPN for VXLAN-over-IPv6
• The ‘skip_config’ node attribute that can be used to deploy partially-provisioned labs
• Lightweight netlab API HTTP server by Craig Johnson
• Rudimentary support for Citrix Netscaler by Seb d’Argoeuves

You’ll find more details (and goodies) in the release notes.

Try to disconnect from the crazy pace of the networking world, forget the “vibe coding with AI will make engineers obsolete” stupidities (hint: Fifth Generation Languages and Natural Language Programming were all the rage in the 1980s and 1990s), and focus on your loved ones. I would also like to wish you all the best in 2026!

In the meantime, I’m working on weaning netlab off of a particular automation tool (you can always track the progress on GitHub). Expect the first results in the January netlab release.

Three releases later (they just released 13.1), the same bug is still there (at least it was on a fresh Python virtual environment install I made on a Ubuntu 24.04 server on December 13th, 2025), making all device_config modules unusable (without changing your Ansible playbooks) for configuration templating. Even worse:

Regardless, you can use underscores in netlab node names (and plugins like multilab use them to create unique hostnames), and they work great on Linux distributions we recommend… until they don’t.

What follows is a story about the weird dependencies that might bite you if you ignore ancient RFCs.

Even though multilevel IS-IS is rarely used today, it always makes sense to understand how things work, and the Multilevel IS-IS Deployments lab exercise created by Dan Partelly gives you a perfect starting point.

Click here to start the lab in your browser using GitHub Codespaces (or set up your own lab infrastructure). After starting the lab environment, change the directory to advanced/1-multilevel and execute netlab up.

Nick Buraglio, one of the working group chairs, provided some answers in Episode 203 of the Software Gone Wild podcast.

Reading that blog post (or the original paper), the inevitable conclusion is that we haven’t made much progress in the last 20 years. Even worse, almost every single pathological architecture described in that blog post applies quite well to real-life organically grown networks.

• Significantly improved Cisco IOS/XR support. With the netlab release 25.12, you can configure VLANs, VRFs, static routes, route redistribution, OSPF default routes, BGP confederations, and BGP local-as
• VXLAN-over-IPv6 on Arista EOS
• VXLAN with ingress replication on Cisco Catalyst 8000v
• The shutdown link/interface attribute can be used to start labs with interfaces turned off
• Large BGP community lists, implemented on Arista EOS, FRR, and Junos. You can use standard- or large community lists in routing policies
• The netlab validate command will reread validation tests from a modified lab topology file every time you run it. It can also read validation tests from a separate file.

• Each VXLAN segment could have a different set of VTEPs (used to build the BUM flooding list)
• While the VXLAN Network Identifier (VNI) must be unique across the participating VTEPs, you could map different VLAN IDs into a single VNI (allowing you to merge two VLAN segments over VXLAN)
• Neither VXLAN VNI nor VLAN ID has to be globally unique (but it helps to make them unique to remain sane)

• Keep it short
• Try to make your point in the first sentence
• Details matter less than you think.

Based on some emails I received in the past (and the lack of response to the lengthy emails I sent), we should apply the same rules to emails (and all other forms of technical communication).

Note: Petr disappeared into the information black hole called Facebook over a decade ago, so I wondered how they allowed him to chat on a podcast for hours. It turns out he moved to NVIDIA, which might influence the podcast content a bit, but I’m pretty sure Petr is still Petr ;)

Note: This is the fifth blog post in the Multi-Pod EVPN series. If you stumbled upon it, start with the design overview and troubleshooting overview posts. More importantly, familiarize yourself with the topology we’ll be using; it’s described in the Multi-Pod EVPN Troubleshooting: Fixing Next Hops.

Ready? Let’s go. Here’s our network topology:

In all podcasts and interviews I listened to, netlab was referred to as a “lab management solution”. But this is misleading. It’s also a translator, due to its ability to abstract devices, and can easily generate perfectly usable configs for devices or technologies you have never worked on.

You might agree with him or not (for example, an accountant or two might get upset with hallucinated invoice items), but his articles are always a fun read.

You’ll find more details (and the opportunity to explore the LSP database contents in a safe environment) in the IS-IS Route Redistribution lab exercise.

Click here to start the lab in your browser using GitHub Codespaces (or set up your own lab infrastructure). After starting the lab environment, change the directory to feature/7-redistribute and execute netlab up.

Let’s see how well Arista EOS is doing; this is their description of the router-id command (taken from EOS 4.35.0F documentation; unchanged for at least a dozen releases):

It’s so nice to see engineers using your tool in real-life scenarios. Thanks a million, Aleksandr, for sharing it.

Note: This is the fourth blog post in the Multi-Pod EVPN series. If you stumbled upon it, start with the design overview and troubleshooting overview posts. More importantly, familiarize yourself with the topology we’ll be using; it’s described in the Multi-Pod EVPN Troubleshooting: Fixing Next Hops.

Ready? Let’s go. Here’s our network topology:

I decided to implement route redistribution (known as route import in netlab) for OSPFv2/OSPFv3, IS-IS, and BGP on IOS/XR (Cisco 8000v running IOS/XR release 24.4.1) and found that each routing protocol uses a different syntax for the source routing protocol part of the redistribute command.

But wait, there’s more: all the lab topologies he used in his exercises are available on GitHub, which means that you could just clone the repo and start using them (I also “borrowed” some of his ideas as future netlab improvements).

Finally, thanks a million to Roman Pomazanov for bringing Dmitry’s work to my attention (and for the quote at the end of his post ;).

If you want to kick the tires, you’ll find the source code on GitHub (Network AI assistant, MCP server for Topolograph service). You’ll also need Vadim’s previous projects: Topolograph and OSPF watcher or IS-IS watcher.

Here’s the lab topology we’ll use (and as usual, the corresponding netlab topology file and device configurations are on GitHub). Our network has two sites (pods), each with a spine switch, a leaf switch, and a host attached to the leaf switch. The inter-pod link is connected to the spine switches to minimize the number of devices.

There’s another batch of snake-oil salesmen consultants peddling their warez to the gullible incompetent managers: the AI preachers promising reduction in support costs. Like the other group of consultants, they have never worked in support and have never implemented a working AI solution in their lives, but that never bothered them or their audience.

Unfortunately, this time I had the unfortunate “privilege” of having the painful front-row seat.

Of course, I wanted to know more about his setup, but it took us over a year and a half till we managed to sit down (virtually) and chat about it, the state of IPv6, the impact of CG-NAT on fraud prevention, and why digital twins don’t make sense in large datacenter migrations.

For more details, listen to Episode 202 of Software Gone Wild.

• SRv6 on IOS/XE. It works with Catalyst 8000v, IOL, and IOL layer-2 image, and can be used to build L3VPNs (the IOS/XE image I have supports no other service on top of SRv6)
• RIPv2/RIPng on OpenBSD
• A more streamlined way to create reports and graphs
• The netlab graph command creates the SVG/PNG/JPEG/PDF graph instead of a graph description file if you’ve installed D2/Graphviz on your system.

We also had to make a few potentially-breaking changes, fixed a bunch of bugs, and added over a dozen small improvements.

You’ll find all the details in the release notes.

What’s amazing is the reality that virtually 100% of tech experts I talk to in the industry feel this way, yet nobody outside of that cohort will mention this reality.

In early September, I received an email from Daniel Blažek telling me that Arista fixed this particular annoyance in the EOS release 4.34.2F. It still uses per-VRF label allocation, but now, you can assign a different label to the default route. Let’s see how that works with our one-arm hub-and-spoke topology:

• Creates an addressing scheme for you
• Designs a routing protocol deployment (OSPF, IS-IS) based on just a few bits of information
• Deploys ready-to-run router configurations to a virtual lab.

You’ll find more details (and the opportunity to tweak the timers in a safe environment) in the Adjust IS-IS Timers lab exercise.

Click here to start the lab in your browser using GitHub Codespaces (or set up your own lab infrastructure). After starting the lab environment, change the directory to feature/6-timers and execute netlab up.

Here’s a quote to get you in the mood:

AI will make good engineers better and will expose mediocre ones. If your value proposition is memorizing CLI commands or being a human grep for log files, then yes, you might need to be worried.

I want to have some validation included in the platform integration tests to ensure the lab devices are started, and that the links and the management network work as expected. The simplest way to get that done is to start OSPF with short hello intervals (to get adjacency up in no time), for example:

We implemented don’t use Ansible release 12 as a quick workaround, but postponing painful things is never a good solution(see also: visiting a dentist), so I decided to try to make netlab work with Ansible release 12. What a mistake to make.

The first lab exercise is already online and expects you to extend a single VLAN segment across an IP underlay network using VXLAN encapsulation with static ingress replication.

The leitmotif: use AI to generate a rough solution, then review and improve it. That makes perfect sense and works as long as we don’t forget we can’t trust AI, assuming you save time doing it this way.

The whole paragraph seems hallucinated², but that couldn’t be because the page was supposedly last updated in 2019, and LLMs weren’t good enough to write well-structured nonsense at that time:

OSPFv3 is backward-compatible with OSPF version 2.

No, it is not.

Adding a Syslog Server

Rsyslog is an open-source implementation of a Syslog server (with many bells and whistles, most of which we won’t use) that can (among other things) log incoming messages to a file. Even better (for our use case), the Rsyslog team regularly publishes Rsyslog containers; we’ll use the rsyslog/rsyslog-collector container because it can “receive logs via UDP, TCP, and optionally RELP, and can send them to storage backends or files.”

OSPF never had such a feature; common implementations set the cost of all interfaces to a very high value to emulate it. BGP got it (the Graceful BGP Session Shutdown) almost 30 years after it was created. IS-IS had the overload bit from day one, and it’s just what an IS-IS router needs to tell everyone else they should stop using it for transit traffic. You can try it out in the Drain Traffic Before Node Maintenance lab exercise.

Click here to start the lab in your browser using GitHub Codespaces (or set up your own lab infrastructure). After starting the lab environment, change the directory to feature/5-drain and execute netlab up.

Before going into the details, let’s get one fact straight: an OSPF router ID was always (at least from the days of OSPFv1 described in RFC 1133) just a 32-bit identifier, not an IPv4 address². Straight from the RFC 1133:

You could use this functionality to include configuration files for Linux containers, custom reports, or even plugins in the lab topology, and share a complete solution as a single file that can be downloaded from a GitHub repository.

Want to check how that feature works with your favorite device? Use the Hide Transit Subnets in IS-IS Networks lab exercise.

Click here to start the lab in your browser using GitHub Codespaces (or set up your own lab infrastructure). After starting the lab environment, change the directory to feature/4-hide-transit and execute netlab up.

• Each fabric is running OSPF and IBGP, with core (spine) devices being route reflectors
• There’s an EBGP session between the WAN edge routers (sometimes called border leaf switches)
• Every BGP session carries IPv4 (underlay) and EVPN (overlay) routes.

In that design, the WAN edge routers have to support EVPN (at least in the control plane) and carry all EVPN routes for both fabrics. Today, we’ll change the design to use simpler WAN edge routers that support only IP forwarding.

Where did all the great documentation go?

In more detail:

There was a time when documentation answered almost all questions:

• What is the thing?
• What does the thing do?
• Why would you use the thing?
• How do you configure the thing?

I’ve seen the same thing happening in training, and here’s my cynical TL&DR answer: because the managers of the documentation/training departments don’t understand the true value of what they’re producing and thus cannot justify a decent budget to make it happen.

However, sometimes you have to apply the same set of commands to several devices. Although you could use device groups to do that, netlab release 25.09 offers a much better mechanism: you can embed custom configuration templates in the lab topology file.

• Support for container version of Cisco 8000v emulator (finally a reasonable IOS-XR platform)
• Support for vJunosEVO (vPTX) release 24+ (it needs UEFI BIOS), thanks to Aleksey Popov and Stefano Sasso
• Wildcards or regular expressions in group- or as_list members.
• Graphing improvements
• OSPFv2/v3 on OpenBSD thanks to Remi Locherer
• OSPFv2/v3 interface parameters on IOS XR

You’ll find more details in the release notes.

How do you handle troubleshooting when VTEPs cannot reach each other across pods?

The ancient Romans already knew the rough answer: divide and conquer.

In this particular case, the “divide” part starts with a simple realization: VXLAN/EVPN is just another application running on top of IP.

Executive Summary: I cleaned up the whole ipSpace.net RSS/Atom feeds system. The script that generated the content for various feeds has been replaced with static Hugo-generated RSS/Atom feeds. I added redirects for all the old stuff I could find (including ioshints.blogspot.com), but I could have missed something. The only defunct feed is the free content feed (which hasn’t changed in a while, anyway), as it required scanning the documents database. You can use this page to find the (ever-increasing) free content.

And now for the real story ;)

Obviously, I wanted to know what drove him to make those changes, what lessons he learned working in various parts of the networking industry, and what (looking back with perfect hindsight) he would have changed.

It looks like they’re trying to solve the how do I connect two containers (network namespaces) without having the privilege to create a vEth pair challenge with plenty of chewing gum and duct tape tap interfaces 🤦‍♂️

Click here to start the lab in your browser using GitHub Codespaces (or set up your own lab infrastructure). After starting the lab environment, change the directory to challenge/01-bird-rr, build the BIRD container with netlab clab build bird if needed, and execute netlab up.

netlab release 25.09 solves that problem with the files plugin and the inline config functionality.

Today, let’s explore a design that will excite the True Believers in end-to-end layer-2 networks: two EVPN fabrics connected with an EBGP session to form a unified, larger EVPN fabric. We’ll use the same “physical” topology as the previous example; the only modification is that the WA-WB link is now part of the underlay IP network.

Let’s see how close we can get to that ideal topology diagram with GraphViz and D2 graphs.

In a 720-degree turnaround, Android 11 supports DHCPv6, but only for prefix delegation purposes. Yes, you got it right, in a year or two, every phone might want to have a dedicated /64 prefix assigned to it on WiFi segments².

Want more details? Well, there’s a high-level overview published on the Android Developers blog and a corresponding message sent to the v6ops mailing list. Let’s see how much sense that makes.

• The hello packets exchanged between routers
• The contents of Link State PDUs flooded across an area or a domain.

Want to know more? Check out the Protect IS-IS Routing Data with MD5 Authentication lab exercise.

Click here to start the lab in your browser using GitHub Codespaces (or set up your own lab infrastructure). After starting the lab environment, change the directory to feature/3-md5 and execute netlab up.

It took me almost a year to get it done, but the functionality finally made it into the 25.09 release:

defaults.device: frr
provider: clab

nodes: [ s1, s2, l1, l2 ]
links: [ s1-l1, s1-l2, s2-l1, s2-l2 ]

This is the graph generated by netlab create followed by dot graph.dot -T png -o graph.png:

The AI overview I got was way too verbose, but it nailed the Key Concepts and How It Works well enough that I could just use them in the netlab README.md file. Maybe this AI thing is becoming useful after all ;)

Click here to start the lab in your browser using GitHub Codespaces (or set up your own lab infrastructure). After starting the lab environment, change the directory to lb/4-ibgp-add-path and execute netlab up.

defaults.device: eos
module: [ bgp ]
nodes:
  a: { bgp.as: 65000 }
  b: { bgp.as: 65001 }

Imagine my astonishment when the two switches failed to configure BGP. Here’s the error message I got when running the netlab’s deploy device configurations Ansible playbook:

For example, what happens if you define a new Routing Control Function (RFC) on Arista EOS and apply it to BGP routing updates in the same configuration session? You’ll find out in the Sorry We Messed Up (video) presentation Stefan Funke had at SwiNOG 40 (note: the bug has been fixed in the meantime).

I wrote an intro to netlab topology graphs years ago, and as expected, it was hopelessly outdated, so I started the project with a complete overhaul of that article.

netlab did not check the Ansible core version (we never had a similar problem in the past), and the installation scripts did not pin the Ansible version (feel free to blame me for this one), which means that any new netlab installation created after September 9th crashed miserably on the simplest lab topologies.

This is the workaround we implemented in netlab release 25.09-post1 (released earlier today):

Want to know more or test the devices you’re usually working with? The Running IS-IS Over Unnumbered/LLA-only Interfaces lab exercise is just what you need.

Click here to start the lab in your browser using GitHub Codespaces (or set up your own lab infrastructure). After starting the lab environment, change the directory to basic/7-unnumbered and execute netlab up.

It turns out I’m too old, and what I know is sometimes no longer true. It seems that the last implementation working as I expected is Cisco IOS Classic ☹️

As expected, our discussion took us all over the place, including (according to Riverside AI):

1. It will be about transceivers
2. It will be fun
3. It will include some crazy stuff

Their SwiNOG 40 presentation (video) met all three expectation. We learned how well transceivers cope with high temperatures and what happens when you try to melt them with a heat gun.

• Link impairment (implemented with Linux netem queuing discipline) defined in lab topology or configured/controlled with the netlab tc command
• Configurable IPv6 Router Advertisement parameters
• The files plugin to store the content of short files (including custom configuration templates) directly in the lab topology
• Support for Nokia SR-OS container (SR-SIM)
• Support for very large topologies (tested so far: approximately 3000 lab devices)

But wait, there’s more (as always):

A few weeks ago, Christian opened an issue describing how netlab breaks when the lab topology has more than 250 devices. We fixed that, only to get into another morass: some code has complexity higher than O(n) (meaning that going from 100 to 200 devices makes things more than twice as slow). Christian is working on one of those problems at the moment (it’s not that his ginormous labs won’t start, it just takes a long time), and I decided it’s time to polish a few other bits of the code.

That blog post used a simple topology with three routers. Now let’s add a few more routers to the mix and see what happens.

We quickly agreed not to go (too deep) into tool-bashing. Instead, we discussed the eternal problems of network automation, from unhealthy obsession with tools to focus on point solutions while lacking the bigger picture or believing in vendor-delivered nirvana.

Their solution uses SRv6 for traffic steering¹, an Intent-Based System² that figures out paths across the network, and eBPF on client hosts³ to add per-application SRv6 headers to outgoing traffic.

That would be interesting by itself (the whole network would appear as a single hub), but they’re also using DCBX (which is riding in LLDP TLVs), and the DCBX parameters are negotiated between servers (not between servers and adjacent switches), sometimes resulting in NIC resets².

• We configure an outbound routing policy to change MED;
• We execute do clear bgp * soft out at the end of most BGP policy configuration templates¹
• The device under test should thus immediately (re)send the expected BGP prefix with the target MED.

That approach failed miserably with ArubaCX; it was time to investigate the details.

netlab has BGP configuration templates for 14 different platforms¹, including these implementations that look like Cisco IOS from a distance if you squint just right²: Arista EOS, Aruba CX, and FRRouting. You can check the configuration templates if you wish; here’s the TC&DB³ overview:

If you have ever watched any of my network automation materials, you won’t be surprised by anything he said, but if you’re just starting your network automation journey, you MUST watch this presentation to get your bearings straight.

Side note: I’m a bit younger than Rodney, but I also went through at least three waves of AI hype cycles, starting with Prolog and 4GL, then expert systems, and finally neural networks. Around that time, I stopped caring and focused on networking, but I have enough battle scars to remain skeptical.

I was improving a completely unrelated BGP functionality. I ran BGP integration tests on Cisco IOL (because it’s the fastest one to boot), and the BGP community propagation test failed. After verifying that I did not change the template and that the data structures had not changed, I checked the IOL release I was using.

Surprise 🎉🎉: the neighbor send-community configurations that worked since (at least) the IOS Classic release 15.x stopped working in Cisco IOS/XE release 17.16.01a.

• Do not panic when the user makes changes to route maps and underlying filters (prefix lists, AS-path access lists, or community lists).
• Let the user decide when they’re done and process the BGP table with the new routing policy at that time.

1. Uses Optimized Ethernet Header, replacing IP/UDP header with a 10-byte something (let’s call it session identifier)
2. Makes Ethernet lossless with hop-by-hop retransmission/error recovery
3. Uses credit-based flow control (the receiver continuously updates the sender about the amount of available space)

If you’re ancient enough, you might recognize #3 as part of Fibre Channel, #2 and #3 as part of IEEE 802.1 LLC2 (used by IBM to implement SNA over Token Ring and Ethernet), and all three as the fundamental ideas of X.25 that Broadcom obviously reinvented at 800 Gbps speeds, proving (yet again) RFC 1925 Rule 11.

I made one of the oldest testing mistakes: I checked whether my test would work under the correct conditions but not whether it would detect an incorrect condition.

• Routing protocols are an eventually-consistent distributed system, and things eventually appear in the right place (if you got the configurations right), but you never know when exactly that will happen.
• You can therefore set some reasonable upper bounds on when things should happen, and declare failure if the timeouts are exceeded. Even then, you’ll get false positives (as in: the test is telling you the configurations are incorrect, when it’s just a device having a bad hair day).

And just when you think you nailed it, you encounter a device that blows your assumptions out of the water.

• The ospf.areas plugin supports OSPFv2 and OSPFv3 stub areas, NSSA areas, and area ranges.
• The BGP routing policies plugin supports aggregate BGP routes
• The BGP configuration module supports BGP confederations

But wait, there’s much more:

• It creates three VRFs (red, blue, and common)
• It imports routes from red and blue VRF into the common VRF and routes from the common VRF into the red and blue VRF (the schoolbook example of common services VRF)
• Just to be on the safe side, it imports red routes into the red VRF and so on.

Here’s the relevant part of the netlab lab topology:

Julia Evans did, and spent months exploring the quirks of the Linux terminal (and writing blog posts describing what she found), finally resulting in The Secret Rules of the Terminal (including the various shells, terminal emulators, escape codes, and TTY driver). A must-read if you’re a newbie who wants to understand why things happen the way they do.

• He’s built a large lab and added tons of extra configuration to the lab devices.
• Afterwards, he realized he’d like to add a few more devices to the lab and was worried about losing all the changes he had made.

Unfortunately, you cannot add new devices to an already-running lab. You must shut down the lab, change the topology description, and start a new lab. However, there are things you can do to preserve the extra work you already did:

As computer systems get more sophisticated we’ve seen a growing trend to value deep specialists. But we’ve found that our most effective colleagues have a skill in spanning many specialties.

Also:

There are two sides to real expertise. The first is the familiar depth: a detailed command of one domain’s inner workings. The second, crucial in our fast-moving field is the ability to learn quickly, spot the fundamentals that run beneath shifting tools and trends, and apply them wherever we land.

Remember how I told you to focus on the fundamentals? 😎

Fortunately, the open-source world doesn’t have to rely on the roadmaps created by networking vendors’ product managers; if there’s a big enough pain, someone will solve it.

I took a trip to the Wireshark land to figure out the details (you can download the capture file):

Fortunately, a few months ago, I spent some time installing the client-side Edgeshark components on my laptop. All I needed to do was enable the edgeshark tool in my lab topology and restart the lab.

Not only was the event impeccably organized (what else would you expect in Switzerland) and at the best event location I have ever experienced (it’s hard to beat this view), it was also full of short, interesting, up-to-the-point presentations (view the slide decks and videos). Plus, I met so many old friends I haven’t seen in years, and people I communicated with for years but never met before.

It’s not like the organizers would need any more publicity (the event was sold out), but if you happen to be near Switzerland in time for the next meeting, make sure to be there.

Thanks again to the wonderful SwiNOG core team for a fantastic experience! I hope we’ll meet again at the next SwiNOG meeting!

Testing the correctness of OSPF configurations seems easy:

• Build a lab with a test device and a few other OSPF devices
• Configure the devices
• Log into the test device and inspect OSPF operational data

There’s just a tiny little fly in this ointment…

OSPFv2 (defined in RFC 2328) is 27 years old, and NSSA functionality (RFC 3101) was last touched 22 years ago. One would hope the implementations in network devices are mature and feature-complete. Yeah, keep dreaming 🤦‍♂️.

A static route has two components: the destination prefix and the next hop – the device that we hope knows how to reach that destination. The next hop is usually specified as an IPv4 or IPv6 address, but may also include outgoing interface information¹.

It’s kind of confusing sometimes to see the digital twin (being a really good idea) never really take off.

His remark prompted me to resurface a two-year-old draft listing a bunch of minor annoyances that make Networking Digital Twins more of a PowerPoint project than a reality.

When I was fixing the errors in netlab SR-OS configuration templates, I couldn’t get the EBGP-based EVPN with overlapping leaf AS numbers to work. I could see the EVPN routes in the SR-OS BGP table, but the device refused to use them. I concluded (incorrectly) that there must be a quirk in the SR-OS EVPN code and moved on.

Before discussing the new features, let’s walk the elephant out of the room: I changed the release versions to YY.MM scheme, so I will never again have to waste my time on the existential question of which number in the release specification to increase.

Now for the new features:

Brief recap: In source routing, the sender has to specify the (loose or strict) path a packet should take across the network. The sender thus needs a mechanism to determine that path, and as always, there are numerous solutions to this challenge. We’ll explore a few of them, using the sample topology shown in the following diagram.

After “fixing” the integration tests to deal with ArubaCX’s notion of VXLAN VNI having 16 bits, the bridging test worked, but the IRB tests kept failing.

In the IRB test, the lab has two layer-3 switches. Each of them should be able to bridge within a VLAN/VXLAN segment and route across the segments.

netlab assumes that most network devices are routers (it considers a firewall to be a router in disguise), apart from Linux hosts, but you can always change what a node is with the role node attribute:

That Ethernet link happens to have the MTU fixed at 1500 bytes. Guess what happens in the world where everyone uses jumbo frames? Did you say fragmentation? Bingo! And what do you think happens when one of those fragments gets dropped due to control-plane policing, and the rest of them are stuck in the reassembly queue? You’ll find the gory details in a lengthy blog post by Nitzan Tzelniker.

Currently, I’m studying for the CCNP ENARSI exam, and would like to start posting my labs to LinkedIn, and perhaps even upload my lab topologies and configs to Git.

That’s a great idea. I would minimize the LinkedIn part¹ and focus on Git:

Usually, whoever is responsible for the contents of the forwarding tables must first discover the network topology. Let’s start there, using the following network diagram to illustrate the discussion.

Before releasing netlab release 2.0, I ran the full suite of integration tests for all devices for which I have the images. Interestingly, most VXLAN tests failed for Cumulus Linux 4.x even though we haven’t touched that code for ages.

Next step: trying to figure out what changed. The configuration changes were minimal. Even worse, the failure was non-deterministic. Somehow, we managed to transform a Cumulus Linux 4.x VM into a Heisenberg switch.

IMHO, the whole NETCONF ecosystem primarily suffers from a tooling problem. Or I haven’t found the right tools yet.

ncclient is (as you mentioned somewhere else) an underdocumented mess. And that undocumented part is not even up to date. The commit hash at the bottom of the docs page is from 2020… I am amazed how so many people got it working well enough to depend on it in their applications.

Now, if only there were a record in a distributed database telling the browser what the web server supports. Oh, wait… Not surprisingly, browser vendors don’t trust that data and have implemented a happy eyeballs-like protocol to decide between HTTPS over TCP and QUIC.

NVIDIA no longer releases Cumulus VX as a standalone image. To simulate a Cumulus Linux switch, use NVIDIA AIR.

And what is NVIDIA AIR?

Each layer can use any forwarding paradigm you choose. However, since we generally use IP at Layer 3, edge devices typically perform hop-by-hop destination-based forwarding, while core devices can use alternative methods.

We decided it’s a good idea to rewrite the VXLAN integration tests to use one target device and one FRR container to test inter-vendor VXLAN interoperability. After all, what could possibly go wrong with a simple encapsulation format that could be described on a single page?

Everything worked fine (as expected), except for the ArubaCX VM (running release Virtual.10.15.1005, build ID AOS-CX:Virtual.10.15.1005:9d92f5caa6b6:202502181604), which failed every single test.

Bonus point: a link to an article by Patrick McKenzie (of the Bits About Money fame) explaining why you SHOULD NOT call yourself a programmer (there goes the everyone should be a programmer gospel 😜).

Using solely the information from blog.ipspace.net, what can you tell me about running ospf over unnumbered interfaces

And then I asked it about unnumbered interfaces and IS-IS, and it all went sideways:

Having read your newest rant around my rant ;-} I can attest that you hit the nail on the very head in basically all you say:

• XML output big? yeah.
• JSON squishy syntax? yeah.
• SSH prioritization? You didn’t live it until you had a customer where a runaway python script generated 800+ XML netconf sessions pumping data ;-)

Thanks a million!

P.S.: If you’ve done something similar and I haven’t noticed it, please send me the link.

FRRouting (Linux) with pure IS-IS, the only way it currently (10.3) works is to copy the loopback IPv4 address to the interfaces that you need to do IPv4 routing on. The OpenFabric (IS-IS “extension” draft) does support true unnumbered interfaces and routes IPv6.

Let’s unpack this. There are (at least) four reasons a router needs an address associated with an interface¹:

• On April 12th, SR Linux failed one of the netlab integration tests. We keep adding functionality to these tests as we discover edge cases we didn’t test before, so sometimes a device that passed the test before might fail the modified version.
• I opened a netlab issue, believing it might be a configuration error on our part.
• It quickly became evident that we’re dealing with an SR Linux bug, as the failure to apply routing policies was random.

I thought that was the end of the story and closed the issue, but then something truly amazing happened:

You could always implement a bridged segment with a set of links connecting edge nodes to a VLAN-capable device. For example, you could use the following topology to connect two Linux hosts through a bridge running Arista EOS:

However, as ChatGPT now has access to the live Internet, I decided to try out whether it can get the job done with a bit of prompting.

TL&DR: After a hiccup, it worked surprisingly well.

I keep saying CLI is an API. However, it is much simpler and an easier way to adapt to the changes, if these three conditions are met:

• Well-defined node roles (host, router, bridge) are now available on multiple platforms
• The firewall.zonebased plugin allows you to configure a rudimentary firewall
• SRv6: BGP L3VPN support is now available for FRRouting, so you can go out and kick its (free) tires.
• bridge nodes can be used as simple bridges or to implement multi-access links
• netlab defaults command provides sysctl-like CLI interface to user/system defaults.

Other changes include:

As always, things were simpler when networks were implemented with a single cable. In that setup, all nodes were directly reachable, and the only challenge was figuring out the destination node’s address; it didn’t matter whether it was a MAC address, an IP address, or a Fiber Channel address. On a single cable, you could just broadcast, like, “Who has this service?” and someone would reply, “I’m the printer you’re looking for.” That’s how many early non-IP protocols operated.

If, on the other hand, the customers would not camp for literally tens of years on regex scripts scraping screens, lots of stuff could progress much faster.

He’s right, particularly from Juniper’s perspective; they were the first vendor to use a data-driven approach to show commands. Unfortunately, we’re still not living in a perfect world:

TL&DR: It doesn’t, unless… ;)

How do A1 and A2 know not to advertise a Type-3 summary LSA generated from area 1 prefixes back into area 1?

He’s right. There is no “originating area” information in the type-3 LSA, so how does an ABR know not to reinsert the type-3 LSA generated by another ABR back into the area?

TL&DR: The OSPF route selection process takes care of that.

A few months later, after your toolchain has been thoroughly tested, you decide to upgrade the operating system on the network devices, and everything breaks. The root cause: the vendor changed their API or the data model between software releases.

I started the lab with the FRRouting routers and configured OSPF area ranges. Astonishingly, I discovered that the more-specific prefixes from an area appear as summary routes in the backbone area even when the area range is configured. When I tried to reproduce the scenario a few days later, it turned out to be a timing quirk (I didn’t wait long enough), but my squirrelly mind was already investigating.

Connecting all relevant devices to a single cable would indubitably simplify any networking stack, but unfortunately, we’re almost never that lucky. We need devices in the network (typically with multiple interfaces) that perform packet forwarding between end nodes.

Even more interestingly, Rudolph points out the elephant I completely missed: RFC 8950 refers to RFC 2545, which requires a GUA IPv6 next hop in BGP updates (well, it uses the SHALL wording, which usually means “troubles ahead”). What do you do if you’re running EBGP on an interface with no global IPv6 addresses? As expected, vendors do different things, resulting in another fun interoperability exercise.

Finally, there’s RFC 7404 that advocates LLA-only infrastructure links, so we might find the answer there. Nope; it doesn’t even acknowledge the problem in the Caveats section.

For even more information, read the Unnumbered IPv4 Interfaces and BGP in Data Center Fabrics blog posts.

TL&DR: default ARP settings on a multi-subnet Linux host are less than optimal.

We use these principles when creating netlab integration tests:

• They should contain a single device-under-test and a bunch of attached probes.
• They should test a single feature.
• They should not rely on the device-under-test. All validation has to be done on probes.

How do you test static routes under these restrictions? Here’s what we did:

Rant mode: ON

This is the (unedited) gem I received after making some of my EVPN videos public:

Not a problem: since Dan Partelly added the netlab exec command, it’s as simple as netlab exec * show ip bgp. Well, not exactly; there are still a few quirks.

Punching holes through that firewall is equivalent to establishing NAT translations.

It would be a shame to let that response wither as small print at the bottom of a blog post; here it is:

You mentioned that only video-conferencing and BitTorrent use client-to-client connectivity (and they are indeed the main use cases), but hell, do they need to engineer complex systems to circumvent these NATs and firewalls: STUN, TURN, ICE, DHT…

Cleaning up the acronym list first: DHT is unlike the others and has nothing to do with NAT.

I even wrote a related integration test, and all our implementations passed it until I asked myself a simple question: “But does it work?” and the number of correct implementations that passed the test without warnings dropped to zero.

I’ve been working in systems engineering (Linux, virtualization, infrastructure ops) and am considering shifting toward network engineering or architecture. I got my CCNA years ago and started CCNP but didn’t continue.

I’d really appreciate any thoughts you might have on how someone with my background could best make that transition today, especially with how things are evolving around automation and the cloud.

I keep answering a variant of this question every other year or so (2019, 2021, 2023, 2024). I guess it’s time for another answer, so here we go.

I encountered a particularly nasty quirk when running the netlab EVPN integration test using symmetric IRB with an anycast gateway between Nokia SR Linux (or Juniper vSwitch) and FRR container.

Multi-homed TCP/IP nodes must have multiple IP addresses because IP uses address interfaces. There is no well-defined procedure in TCP/IP for how a multi-homed node should behave. In the early days of TCP/IP, they tried to address that in RFC 1122 (Host Requirements RFC), but even then, there were two ideas about dealing with multiple interfaces: the strong and weak end system models (more details).

It’s a must-read, even if you’re absolutely uninterested in the topic. Just replace “Quantum mumbo-jumbo” with AI or SDN ;) Have fun!

In a more controlled environment, you’d want to define BGP neighbors on the BGP RR/RS but not waste CPU cycles trying to establish BGP sessions with unreachable neighbors. Welcome to the world of passive BGP sessions.

Click here to start the lab in your browser using GitHub Codespaces (or set up your own lab infrastructure). After starting the lab environment, change the directory to session/8-passive and execute netlab up.

From my young millennial point of view, the logic is reversed: it is because of NATs and firewalls that the internet became so asymmetrical (client/server) just like the Minitel was designed (yes, I am French), whereas the Internet (and later the web, although a client/server protocol, was meant for everyone to be a client and a server) was designed to be more balanced.

Let’s start with the early Internet. It had no peer-to-peer applications. It connected a few large computers (mainframes) that could act as servers but also allowed terminal-based user access and thus ran per-user clients.

Zvi Mowshowitz collected links to over a dozen different writing styles, starting with JRR Tolkien. I’m pretty sure you’ll find something useful in that vast collection.

• Add default gateway (including anycast- and VRRP gateway )as a valid next-hop for static routes
• Rewrite the default gateway processing and add IPv6 default gateways on links without anycast or VRRP gateways
• Set libvirt MTU to 9500 on bridge-based networks to avoid the “transparent fragmentation” on Linux bridges.
• Use device- or node variables to specify the Juniper vMX license file.

I wanted to test whether we got the nasty nuances of VRRPv3 IPv6 configuration right on all supported platforms and created a simple lab topology in which the device-under-test and an Arista cEOS container would be connected to two IPv6 networks (Arista EOS is a lovely device to use when testing a VRRP cluster because it produces JSON-formatted show vrrp printouts).

Most platforms worked as expected, but Aruba CX, Cumulus Linux with NVUE, and Dell OS10 consistently failed the tests. We were stumped until Jeroen van Bemmel discovered that the Arista container forwards IPv6 router advertisements between the two LAN segments.

• All interfaces are connected to a VLAN-aware switch
• The switch interfaces could be access or trunk interfaces¹.
• Each VLAN in a VLAN-aware switch can be connected to an internal router through a VLAN interface.

However, that’s not how we configure layer-3 switches. There’s a significant gap between the conceptual configuration model and the internal architecture:

While I’m not exactly an OSI fan, I must admit they got many things right (and IPv6 copied those ideas), but TCP/IP is a clear winner in this aspect.

In the meantime, those sequences got standardized and (par for the course) extended with “proprietary” stuff everyone uses now. Julia Evans did a great job documenting the state of the art. Thanks a million!

It started innocently enough: I was working on the VRRP integration test and wanted to use Arista EOS as the second (probe) device in the VRRP cluster because it produces nice JSON-formatted results that are easy to use in validation tests.

Everything looked great until I ran the test on all platforms on which netlab configures VRRP, and all of them passed apart from Arista EOS (that was before we figured out how Sturgeon’s Law applies to VRRPv3) – a “That’s funny” moment that was directly responsible for me wasting a few hours chasing white rabbits down this trail.

Want to know more? The Influence the Designated IS Election lab exercise provides the details (and some hands-on work).

As the first step, I created a simple netlab topology in which a single link had a slightly larger than usual MTU… and then all hell broke loose.

Unfortunately, you’re only half done once you have installed the LSPs. You still have to persuade the network devices to use them. Welcome to the confusing world of traffic steering explored in the Loopback as a Service blog post by Dmytro Shypovalov.

Cumulus Linux (NVUE):

• VXLAN and EVPN
• VLAN-aware router (VLAN subinterfaces) functionality
• VRF route leaking
• VRF-aware BGP and full RFC 8950 support (IPv4 BGP AF over regular IPv6 BGP session)
• BGP allowas_in and EBGP multihop

While network devices provide traffic capture capabilities (usually tcpdump in disguise generating a .pcap file), it’s often better to capture the traffic outside of the device to see what the root cause of the problems you’re experiencing might be.

However, it turned out one can have a fun conversation about a controversial topic. For more details, listen to The Rise of NAT on Chaos Lever. I hope you’ll enjoy it ;)

• Is the transition to IPv6 inevitable? (hint: Betteridge’s law of headlines)
• The mistakes and missed opportunities in the design of IPv6 - episode 1 (aka Second System Effect¹)

I’m pretty sure Ole won’t stop there, so stay tuned.

However, sometimes we need stub segments – segments connected to a single router or switch – because we don’t want to waste resources creating hosts attached to a network device, but would still prefer a more realistic mechanism than static routes to inject IP subnets into routing protocols.

• Part 1 covers the ancient Netflow v5, Netflow v9, and IPFIX. It also mentions sampling and flow aggregation.
• Part 2 describes sFlow, port mirroring and sampled mirroring, and the use of IPFIX/Netflow v9 to transport mirrored traffic.

These blog posts will not make you an expert but will give you an excellent overview of the telemetry landscape¹.

If you haven’t set up your own lab infrastructure, click here to start the lab in your browser using GitHub Codespaces. After starting your codespace, change the directory to basic/e-ebgp-multihop and execute netlab up.

• The reasons people might still prefer RSVP-TE over SR-MPLS and the current state of RSVP-TE
• What an SDN controller might bring to the RSVP-TE world
• SR/RSVP coexistence and interworking

Have fun!

Fortunately, Vagrant debugging is pretty good¹ and I was quickly able to pinpoint the issue (full printout):

If you were looking for a nice excuse to visit that part of Europe (it’s been on my wish list for a very long time), this might be a perfect opportunity to do it 😎.

On a tangential topic of fascinating destinations 😉, there’s also ITNOG in Bologna (May 19th-20th, 2025), Autocon in Prague (May 26th-30th, 2025), and SWINOG in Bern (late June 2025).

For example, these are partial test results from the OSPFv2 tests:

Network Automation with Python

Project: Automate basic network tasks like device configuration, backup, or monitoring using Python scripts.

In this blog post, we’ll see how KVM/QEMU/libvirt/Vagrant use UDP tunnels to connect virtual machines, and how containerlab creates point-to-point vEth links between Linux containers.

At that point, I should have said, “that’s crazy, we shouldn’t allow that” and enforce the “VLAN 1 has to be used as a native VLAN” rule. Alas, 20/20 hindsight never helped anyone.

TL&DR: Do not use VLAN 1 in VLAN trunks; if you have to, use it as a native VLAN.

Imagine you have a large topology with dozens of links, and you get an error saying, “there is this problem with links[17]”. It must be great fun counting the links to find which one triggered the error, right?

I would love to figure out what takes them so long (a minute is an eternity on modern CPUs), but I guess we’ll never know.

Behind the Scenes

netlab uses two device provisioning mechanisms: it can start virtual machines with Vagrant or containers with containerlab. Some of those containers might use KVM/QEMU to run a hidden virtual machine (see also: RFC 1925 rule 6a).

TL&DR: YES 🎉 🎉

Here’s what you have to do to make the Arista cEOS container work with netlab running on an Ubuntu VM on Apple silicon:

• Point-to-point links between devices (usually using some variant of Ethernet)
• Multi-access layer-1 networks running some IEEE 802.x encapsulation on top of that (GPON, WiFi, Ethernet hubs)
• Multi-access switched layer-2 network (dumb switches, hopefully running some STP variant)

Implementing these connections in virtual labs is a bit harder than one might think, as all virtualization solutions assume you plan to run virtual servers connected to Ethernet segments.

netlab release 1.9.4 passed hundreds of integration tests and should be a better choice than the previous 1.9 releases. To upgrade, execute pip3 install --upgrade networklab.

Update: 2025-02-03

We still missed a few quirks :( Release 1.9.4-post1 addresses those (and, unfortunately, I’m pretty sure there will be more).

Have you ever seen a BGP peer in the “Connect” state? In 20 years, I have never been able to see or reproduce this state, nor any mention in a debug/log. I am starting to believe that all the documentation is BS, and this does not exist.

The BGP Finite State Machine (FSM) (at least the one defined in RFC 4271 and amended in RFC 9687) is “a bit” hard to grasp but the basics haven’t changed from the ancient days of RFC 1771:

First, your statement on Autonetkit is indeed correct. We had removed that from the product due to lack of popularity. That being said, in our roadmap we are looking at methods to reintroduce on-the-fly configuration as well as enhancing our sample labs library to make getting started with CML easier.

Secondly, CML can be run in full IaC mode because of the API-first build. In fact, many of our customers are using CML as an automated test/validation bed for their CI/CD pipelines. Tools like Ansible and Terraform are available to facilitate this inside CML too. For more details, read:

• Get Started With Terraform and Cisco Modeling Labs
• How to Use Ansible with CML

It pretty much mirrors my experience (plus, I got annoyed when the semi-relevant suggestions kept kicking me out of the flow) and reminds me of the early days of OpenFlow, when nobody wanted to listen to old grunts like myself telling the world it was all hype and little substance.

This is how you configure the basic VRRPv3 parameters for IPv4 on a Cisco IOS/XE device:

interface GigabitEthernet0/1
  vrrp 217 address-family ipv4
    address 172.16.33.42

You would expect something similar for IPv6, right? You’d be right if you were working with Arista EOS:

• The BGP neighbor has to send information about all prefixes in its BGP table
• The device with an inbound filter wastes additional CPU cycles to drop many incoming updates.

Wouldn’t it be better for the device with an inbound filter to push that filter to its BGP neighbors?

The idea was pretty simple:

• Create a lab with the tested device and a well-known probe connected to the same subnet.
• Disable VRRP (or interface) on the probe and check IPv4 and IPv6 connectivity through the tested device (verifying it takes over ownership of VRRP MAC and IP addresses).
• Reenable VRRP on the probe and change its VRRP priority several times to check the state transitions through INIT/BACKUP(lower priority)/MASTER(change in priority)/BACKUP(preempting after a change in priority).

People steeped in a slightly more nuanced view of the world in which IP is not the centerpiece of the universe might tell you that the blob of stuff we need is two things:

TL&DR: Of course, but that does not mean you should.

Anyway, leaving behind the land of sane designs, let’s trot down the rabbit trail of IBGP-only networks.

A bit of a background first:

• In the end, netlab collects all links in the links list before starting the data transformation process.
• Every entry in the links list is a dictionary. That dictionary can contain link attributes and must contain a list of interfaces connected to the link.
• Every interface must have a node (specifying the lab device it belongs to) and could contain additional interface attributes.

What does that mean, and why does it matter? You’ll find the answers in the Optimize Simple IS-IS Deployments lab exercise.

iBGP designs would require route reflectors and additional processing, which could result in slightly slower convergence.

Let’s see whether that claim makes any sense.

TL&DR: No. If you’re building a simple leaf-and-spine fabric, the choice of the routing protocol does not matter (but you already knew that if you read this blog).

Today’s special: Junos is failing the IS-IS metrics test.

The test is trivial:

• The device under test is connected to two IS-IS routers (X1 and X2)
• It has a low metric configured on the link with X1 and a high metric configured on the link with X2

The validation process is equally trivial:

• The lab should have two sites (A and B).
• Each site has a layer-3 switch, a single VLAN (VLAN 100), and two hosts connected to that VLAN.
• As you don’t believe in the magic powers of stretched VLANs, you have a layer-3 (IPv4) link between sites.

With the new documentation prefix (3fff::/20) (defined in RFC 9637), there’s absolutely no excuse to use public IPv6 address space in examples anymore.

• Multi-chassis Link Aggregation (MLAG) on Arista EOS, Aruba CX, Cumulus NVUE, and Dell OS10
• VRF and VLAN groups
• Additional OSPF interface parameters (hello and dead timers, cleartext passwords, and DR priority) implemented on Arista EOS, Aruba CX, Cisco IOS/IOS-XE, Cisco Nexus OS, Cumulus Linux, Dell OS10, and FRRouting
• Static routes with direct or indirect next hops implemented on Arista EOS, Cisco IOS/IOS-XE, FRRouting, and Linux
• Node cloning plugin for users who want to build detailed digital twins of their networks.
• Consistent selection of default address pools based on the number of nodes attached to a link (this could change addressing in multi-provider topologies)
• Support for vjunos-router and Cisco NSO tool.

Other new features include:

Let’s start with:

Late last year, I decided to run netlab integration tests with the Bird routing daemon. It passed most baseline netlab OSPFv3 integration tests but failed those that checked the loopback IPv6 prefix advertised by the tested device (test results).

I hope you’ll also be able to disconnect from the crazy pace of the networking world, forget the “AI will make networking engineers obsolete” shenanigans (hint: SDN did not), and focus on your loved ones. I would also like to wish you all the best in 2025!

I will probably get bored sometime in late December, so expect a few new netlab features in early January.

I particularly liked the “‌AI is like having a very eager junior developer on your team” take, as well as the description of the “70% problem” (AI will get you 70% there, but the last 30% will be frustrating) – a phenomenon perfectly illustrated by the following diagram by Forrest Brazeal:

Click here to start the lab in your browser using GitHub Codespaces (or set up your own lab infrastructure). After starting the lab environment, change the directory to policy/b-disaggregate and execute netlab up.

Hope you’ll find it useful ;)

TL&DR: No, it does not. You were just lucky.

Basics first¹. BGP runs over TCP sessions. One of the first things a router does when establishing a BGP session with a configured neighbor is to open a TCP session with the configured neighbor’s IP address.

However, if you want to know the behind-the-scenes details, you MUST read the Codespaces for Network Engineers and Educators deep dive by Julio Perez.

Terminology first:

• Prefix Independent Convergence allows entries in the forwarding table to point to shared next hops (or next-hop groups), reducing the FIB update bottleneck when changing the next hop for a large number of prefixes (for example, when dealing with a core link failure). More details in the initial blog post and PIC applicability to fast reroute.
• PIC Edge (as defined by vendor marketing) is the ability to switch to a backup CE route advertised to a backup PE router before the network convergence is complete.

Here’s (in a nutshell) how PIC Edge is supposed to work:

We don’t have anything similar in the netlab examples yet, so let’s build a simple topology with two routers, a firewall, and a few hosts.

However, we have to start with a few caveats:

He didn’t post the underlying topology file, so let’s create a simple topology to build something similar.

Want to know more? You’ll find the details in the Dual-Stack (IPv4+IPv6) IS-IS Routing lab exercise.

• I was ranting about that problem in 2009 (15 years ago) and did a summary of older rants in 2015.
• It was evident to everyone but the religious zealots that the only solution we have at the moment is either NAT (because stuff simply does not work otherwise) or host-based solutions that never got implemented (apart from a few rare cases of multipath TCP).

Anyway, Fernando wraps up his draft with:

There’s just a tiny gotcha in the above Works Best in PowerPoint diagram. IBGP assumes the BGP neighbors are in the same autonomous system while EBGP assumes they are in different autonomous systems. The usual way out of that OMG, I painted myself into a corner situation is to use BGP local AS functionality on the underlay EBGP session:

Click here to start the lab in your browser using GitHub Codespaces (or set up your own lab infrastructure). After starting the lab environment, change the directory to session/9-dynamic and execute netlab up.

Want to know more? The Using IS-IS Metrics lab exercise provides all the gory details.

Thanks to Networking Notes, I found a great infographic I can use in my next presentation (bonus points: it also works great in a terminal when fetched with curl) and a site that checks the latency of your web site from various vantage points.

BGP route servers are commonly used on Internet Exchange Points (IXPs), and that’s what you can practice in the BGP Route Server in an Internet Exchange Point lab exercise.

Click here to start the lab in your browser using GitHub Codespaces (or set up your own lab infrastructure). After starting the lab environment, change the directory to session/5-routeserver and execute netlab up.

Things behave really badly if the total IGP cost over the tunnel undermines the total topologies’ cost. What happens next is that the tunnel “wraps” around itself, ultimately causing a meltdown of the entire network.

Let’s unpack that, starting with “Why would you need a tunnel?”

• Jeroen van Bemmel added the spanning tree and link aggregation configuration modules, initially implemented on Arista EOS, Cumulus Linux, and FRR.
• Dan Partelly added the netlab exec command that can execute the same command on a set of network devices, support for Edgeshark, and support for Cisco IOS on Linux (IOL) and Cisco IOS layer-2 image on Linux (IOLL2), the latter after a heroic uphill battle with ancient software (part 1, part 2).

Other new features include:

Oh, and please use Hugo (or similar) and use walled gardens like LinkedIn solely to post summaries and links to your content. You want to be in control and retain ownership of your work, right?

TL&DR: I loved the experience ;)

Now we’re entering Wonderland: the somewhat unusual¹ things vendors do to make their existing stuff work while also pretending to look cool². We’ll start with EBGP-over-EBGP, and to understand why someone would want to do something like that, we have to go back to the basics.

If the VLAN is an IRB VLAN (which can be modified globally or per node with the VLAN mode parameter), netlab also creates the VLAN (or SVI, or BVI) interface. But how do you specify the parameters of the VLAN interface?

TL&DR: It doesn’t. However…

netlab is a CLI command that acts as an umbrella orchestration layer for Vagrant and Containerlab. It does not run as a cron job, init script, or service and thus cannot be invoked when a server is booted.

At that time, I had lived in a comfortable Cisco IOS bubble for way too long, so my answer was along the lines of “Say what???” Nicola Modena⁶ quickly expanded my horizons, and I said, “Gee, I have to write a blog post about that!” As you can see, it took me over a decade.

That’s what you’ll do in today’s lab exercise, which also explains the behind-the-scenes differences between point-to-point and multi-access links and the intricate world of three-way handshake.

Also, it was such a nice experience ;)

Let’s start with the elephant in the room: how do you know whether you can reach a host you want to communicate with directly? In the following diagram, how does A know whether B is sitting next to it?

• Poor Man’s Traffic Engineering
• Egress Peer Engineering: Basics
• Egress Peer Engineering: Building Blocks

Have fun!

As you probably know by now, there’s always more than one way to get something done with BGP. Today, we’ll explore how you can use the NO_EXPORT community to filter transit routes.

Click here to start the lab in your browser using GitHub Codespaces (or set up your own lab infrastructure). After starting the lab environment, change the directory to policy/d-no-export and execute netlab up.

The episode was published a few days ago; I hope you’ll enjoy listening to it.

Now, imagine someone persuaded you that EBGP is better than any IGP (OSPF or IS-IS) when building a data center fabric. You’re running EBGP sessions between the leaf- and the spine switches and exchanging IPv4 and IPv6 prefixes over those EBGP sessions. Can you use the same EBGP sessions for EVPN?

TL&DR: It depends™.

• IS-IS route redistribution and IS-IS VRF instances
• RIPv2/Ripng route redistribution and VRF instances
• Configurable RIPv2/RIPng protocol timers

We also added support for Cisco IOSv layer-2 image. You’ll find more details in the release notes.

Big things first. I added validation to these labs:

• MD5 Passwords and GTSM
• Use BGP Timers and BFD to Speed Up BGP Convergence
• BGP Route Aggregation
• EBGP Sessions over IPv6 LLA Interfaces
• Establish an IBGP Session
• Build a Transit Network with IBGP
• Use BGP Route Reflectors
• Using No-Export Community to Filter Transit Routes

I also tried to make the IS-IS labs more than just lab exercises. Each exercise includes a bit of background information or IS-IS theory; this one describes generic OSI addresses (NSAPs) and router addresses (NETs).

TL&DR: YES, IT WORKS 🎉 🎉

Here’s what you have to do to make SR Linux work with netlab running on a Ubuntu VM on Apple silicon:

TL&DR: Yes, but only on some devices (for example, Cisco IOS or FRRouting) when using MPLS transport.

Here’s a high-level diagram of what we’d like to achieve:

It is a bit sad, though, that we still need “how to use IPv6” books when the protocol is old enough to enjoy a nice glass of whiskey (in the US) trying to drown its sorrow at its slow adoption.

Let’s create the report I used in the EVPN Hub-and-Spoke Layer-3 VPN blog post to create the VRF table.

• We’ll split the single PE router into three PE devices (pe_a, pe_b, and pe_h)
• We’ll add a core router (p) and connect it with all three PE devices.

As we want to use EVPN and have a larger core network, we’ll also have to enable VLANs, VXLAN, BGP, and OSPF on the PE devices.

This is the topology of our expanded lab:

You get the following diagram when you model the traffic flow requirements with VRFs. The forward traffic uses light yellow arrows, and the return traffic uses dark orange ones.

Want an easy start? Use GitHub Codespaces. Have a laptop with Apple Silicon? We have you covered ;)

As before, this is the fabric we’re working with:

1. Routes received from BGP neighbors are stored in BGP RIB.
2. Routes redistributed into BGP from other protocols are (re)created in the BGP RIB.
3. BGP selects the best routes in BGP RIB using its convoluted set of rules.
4. Best routes from the BGP RIB are advertised to BGP neighbors
5. Best routes from the BGP RIB compete (based on their administrative distance) against routes from other routing protocols to enter the IP routing table (system RIB)
6. Routes from the system RIB are copied into FIB after their next hops are fully evaluated (a process that might involve multiple recursive lookups).

The reason of L2VPN is becoming more popular by service providers and customers is about provisioning complexity.

The book seems to be in an early (ever-evolving) stage, but it’s well worth exploring if you’re new to the IPv6 world, and you might consider contributing if you’re a seasoned old-timer.

It would also be nice to have a few online labs to go with it ;)

During the summer, he published some of them in a collection of GitHub repositories and made them work in GitHub Codespaces. An amazing idea well worth exploring!

TL&DR: It works (on Arista cEOS)¹.

Here are the relevant parts of a netlab lab topology I used in my test (you can find the complete lab topology in netlab-examples GitHub repository):

• Bridging multiple VLANs
• Asymmetric IRB, symmetric IRB, central routing, and running OSPF within an IRB VRF.
• Layer-3 only VPN, including routing protocols (OSPF and BGP) between PE-router and CE-routers
• All designs evangelized by the vendors: IBGP+OSPF, EBGP-only (including reusing BGP AS number on leaves), EBGP over the interface (unnumbered) BGP sessions, IBGP-over-EBGP, and EBGP-over-EBGP.

All tests included one or two devices under test and one or more FRR containers¹ running EVPN/VXLAN with the devices under test. The results were phenomenal; apart from a few exceptions, everything Just Worked™️.

There are several benefits of SRv6 that I’ve heard of.

Want to try out that concept? Start the Using No-Export Community to Filter Transit Routes lab in GitHub Codespaces.

Interestingly, at least some EVPN implementations handle multiple VNIs per VRF without a hitch; I ran my tests in a lab where three switches used unique per-switch VNI for a common VRF.

I could solve the problem in a few ways:

• Generic Routing Configuration Module implements routing policies (route maps), prefix filters, AS-path filters, and BGP community filters.
• Default route origination in OSPFv2 and OSPFv3
• Route import (redistribution) into OSPFv2, OSPFv3, and BGP.
• Named prefixes

Other new goodies include:

A Bit of Theory

BGP runs over TCP, and all parameters related to the TCP session are configured for a BGP neighbor (IPv4 or IPv6 address). That includes the source interface, local AS number (it’s advertised in the per-session OPEN message that negotiates the address families), MD5 password (it uses MD5 checksum of TCP packets), GTSM (it uses the IP TTL field), or EBGP multihop (it increases the IP TTL field).

After a few configuration tweaks and a batch of integration tests later, I had the results: everything worked. You can use MPLS on Arista cEOS with netlab release 1.9.0 (right now @ 1.9.0-dev2), and I’ll be able to create MPLS labs running in GitHub Codespaces in the not-too-distant future.

The most important thing to understand when analyzing a layer-3-only EVPN/VXLAN network is that the data plane looks like a VRF-lite design: each VRF uses a hidden VLAN (implemented with VXLAN) as the transport VLAN between the PE devices.

The benefit of VXLAN is mostly scalability, so if your enterprise network is not scaling… just don’t. The migration path from VLANs is to just keep using VLANs. The (vendor-driven) networking industry has a huge blind spot about this.

Paraphrasing the famous Dinesh Dutt’s Autocon1 remark: I couldn’t disagree with you more.

The reality is a bit muddier (in the VXLAN world) as we still need transit VLANs and router MAC addresses; the best way to explore what’s going on behind the scenes is to build a simple lab.

The majority of the networks running now in the Enterprise are on traditional VLANs, and the migration paths are limited. Really limited. How will a business transition from traditional to whatever is next?

The only sane choice I found so far in the data center environment (and I know it has been embraced by many organizations facing that conundrum) is to build a parallel fabric (preferably when the organization is doing a server refresh) and connect the new fabric with the old one with a layer-3 link (in the ideal world) or an MLAG link bundle.

I agree entirely with his conclusions (avoid both features). However, I still think that replacing an AS within the confederation part of an AS path (which should belong to a single well-managed AS) is not exactly the most brilliant idea I’ve seen.

You can generate the reports in text, Markdown, or HTML format. The desired format is selected with the report name suffix. For example, the bgp-asn.md report will create Markdown text.

Let’s see how that works.

I’m curious what EVPN provides if all you need is L3. For example, you could run pure L3 BGP fabric if you don’t need VRFs or a limited amount of them. If many VRFs are needed, there is MPLS/VPN, SR-MPLS, and SRv6.

I received a similar question numerous times in my previous life as a consultant. It’s usually caused by vendor marketing polluting PowerPoint slide decks with acronyms without explaining the fundamentals¹. Let’s fix that.

Do you think, realistically in 2024, netlab would suffice to prepare the CCIE lab exam? Particulary for the SP flavor, since netlab supports a lot of routing protocols. Thanks!

TL&DR: No.

netlab would be a great tool to streamline your CCIE preparation studies. You could:

The real danger today is not that computers are smarter than us but that we think computers are smarter than us and consequently trust them to make decisions they should not be trusted to make.

It might be worth remembering that quote when an AI-powered management appliance messes up your network because of a false positive ;)

He also succinctly summarized everything I ever wanted to say about the idea of using AI tools to generate networking configurations:

Having a tool that makes stuff that looks right but ends up broken is worse than not having the tool at all.

You should also read the follow-up post: what happens if a VLAN-unaware switch receives an 802.1Q-tagged frame?

MPLS worked – and it still works – but it provided optimal value in an earlier time when the center of gravity was not the cloud. The cloud challenged the efficacy of MPLS, and it wasn’t long before SD-WAN, cloud connects, and interconnects […] represented an implacable threat to a status quo that had once seemed unassailable.

The second part of the paragraph is (almost) true, but it had nothing to do with MPLS.

You’ll also figure out:

• Why does CTRL-A jump to the beginning of the line?
• How can you enable command line editing in ancient utilities?

Have fun!

• Twenty-six deploy BGP exercises, including advanced settings like AS path manipulations, MD5 passwords and BFD, and new technologies like TCP/AO and interface EBGP sessions.
• Fifteen BGP routing policies exercises, covering the basic mechanisms as well as dirty tricks like route disaggregation
• Four load balancing exercises, from EBGP ECMP to BGP Link Bandwidth and BGP Additional Paths.
• Five challenges for everyone who got bored doing the simple stuff ;)

That completes the BGP technologies I wanted to cover. I’ll keep adding the challenge labs and advanced scenarios. Here are some ideas; if you have others, please leave a comment.

Apart from having the satisfaction of ticking off a long-outstanding project, the blog pages should load faster, and I won’t have to deal with GitLab hiccups anymore.

If you notice anything being broken, please let me know. Thank you!

Because these programs cannot themselves be concerned with truth, and because they are designed to produce text that looks truth-apt without any actual concern for truth, it seems appropriate to call their outputs bullshit.

Have fun!

I did attempt some of your labs, like IPv6 link-local-only BGP with FRR hosts, but FRR seemed not to play ball, or I was just doing it wrong.

As he was already using netlab, I could send him a cheat code:

Yesterday, I explained how you can run netlab examples in GitHub codespaces and mentioned that they work best with vendors who understand the value of frictionless downloads. But what if you’d like to use a device from one of the good guys who provide the container images but require a registration?

It turns out the solution is trivial:

Setting that up was easy: copy the .devcontainer directory from the BGP labs repository into the netlab examples repository and commit the change. After a brief yak-shaving exercise (writing README files and rearranging a few folders), I successfully started the codespace and was ready for this blog post. There was just one gotcha…

If you agree with me that that’s not necessarily the best idea out there, you might enjoy this rant by Nikhil Suresh.

Want to know more? You can explore a real-life BGP wedgie and fix it in the latest BGP lab exercise.

1. Running BGP between VMs and central BGP route servers
2. Using multihop EBGP session to send full Internet routing table to a customer without overloading the PE-router
3. Running EBGP EVPN session between loopbacks advertised with EBGP IPv4 session (🤢)

However, I would love to start with a simple use case to help engineers unfamiliar with BGP realize when they might have to use multihop EBGP sessions. Unfortunately, I can’t find one, and the scenarios where I used multihop EBGP in the past (EBGP load balancing and using a low-end router in the EBGP path, where I was effectively using the reverse application of #2 as a customer) are mostly irrelevant.

Would you have an easy-to-understand use case that is best solved with a multihop EBGP session? Please share it in the comments. Thanks a million!

Not too long ago, the containerlab team launched the ability to run containerlab within a free¹ container² running on GitHub, and that seemed like a perfect solution to run the BGP labs (Jeroen van Bemmel pointing me in the right direction was another significant step forward).

Someone tried to use netlab with vrnetlab containers for CSR 1000v and Nexus 9300v. We got it to work, but when I started integrating his changes into the development branch, I wanted to test them, so I installed vrnetlab to create my own container images. vrnetlab is an excellent tool, and building containers is a breeze (running them is a different story), so I added support for vrnetlab containers for every device supported by that tool and netlab for which I happened to have a disk image.

I would suggest providing netlab validate for each lab.

Numerous rounds of yak-shaving later, I merged a humongous commit that adds automated validation to these lab exercises:

Other new features in netlab release 1.8.3 include support for BGP route servers (and route server clients), BGP Link Bandwidth community, and OSPF/BGP validation plugins for Arista EOS, Cumulus Linux and FRR. We also fixed the installation scripts to work with Ubuntu 24.04 and Debian Bookworm.

For more details, read the release notes.

When you’re dealing with a large cluster on a fabric, you will see things like inband backup. The most common one I’ve seen is VEEAM. Those inband backups can flood a single link, and no amount of link scheduling really solves that; depending on the source, they can saturate 100G. There are a couple of solutions; IPv6 or eBGP SID has been used to avoid these links or schedule avoidance for other traffic.

It is true that (A) in-band backups can be bandwidth intensive and that (B) well-written applications can saturate 100G server links. However:

Keep that in mind the next time someone tries to sell you the beauties of a centralized control plane – an idea that should be dead by now regardless of what ONF keeps preaching but will inevitably reappear in some form or other due to RFC 1925 Rule 11.

A central hub to manage the data, templates and playbooks that powers your infrastructure by combining the version control and branch management capabilities of Git with the flexible data model and UI of a graph database.

I’ve seen an early demo, and it looks highly promising and absolutely worth exploring. Have fun ;)

As Gerben Wierda explains in his recent “When ChatGPT summarises, it actually does nothing of the kind” blog post, you have to understand a text if you want to generate a useful summary, and that’s not what LLMs do. They can generate a shorter version of the text, which might not focus on the significant bits.

The same lesson applies to network design: cramming too many features into a single device will inevitably result in complex, hard-to-understand configurations and weird bugs. Sometimes, it’s cheaper to split the required functionality across multiple devices.

There are four (increasingly complex) questions our tests should answer:

For more details, read AWS: Egress Traffic and Using AWS Services via IPv6 (rendered in beautiful, easy-to-read teletype font).

The transformation code (business logic) is one of the most complex pieces of a network automation solution, and there’s only one way to ensure it works properly: you test the heck out of it ;) Let me show you how we solved that challenge in netlab.

Steve Deering presented results for the Dijkstra calculation in the “MOSPF meeting report” in [3]. Steve’s calculation was done on a DEC 5000 (10 mips processor), using the Stanford internet as a model. His graphs are based on numbers of networks, not number of routers. However, if we extrapolate that the ratio of routers to networks remains the same, the time to run Dijkstra for 200 routers in Steve’s implementation was around 15 milliseconds.

Let’s add a bit of complexity and replace a traditional layer-2 fabric with a VXLAN fabric. The MLAG cluster members still use an MLAG peer link and an anycast VTEP IP address (more details).

• Check for Vagrant boxes or Docker containers before starting the lab and display pointers to build recipes.
• Check installed Ansible collections before trying to configure the lab devices.
• Display a warning if the lab topology was modified after the lab was created

TL&DR: If you use the same VRF on multiple devices, it’s better to define it globally.

However, you might not need every VRF on every lab device in a more complex lab topology. Considering that, netlab tries to minimize the number of VRFs configured on lab devices using a simple rule: a VRF is configured on a lab device only if the device has at least one interface in that VRF.

In this blog post, we’ll focus on traditional MLAG cluster implementations using a peer link; another blog post will explore the implications of using VXLAN and EVPN to implement MLAG clusters.

We’ll also ignore the interesting question of “how is the LAG member link failure detected?”¹ and focus on “what happens next?” using the sample MLAG topology:

Obviously, the concept is nothing new. I wrote about a similar problem in xDSL networks in 2009.

Nice to hear the Cisco intends to support LISP. However, it is removed from IOS XR already. So it is not that clear…

If Cisco will stop supporting LISP, then we will be forced to create our own LISP routers, since we need it for extreme mobility environments.

However, those virtual machines better work out of the box, or you’ll get frustrated engineers who will give up and never touch your warez again, or as someone said in a LinkedIn comment to my blog post describing how Junos vPTX consistently rejects its DHCP-assigned IP address: “If I had encountered an issue like this before seeing Ivan’s post, I would have definitely concluded that I am doing it wrong.”²

Let’s hope he will keep them coming and link them together so it will be easy to find the whole series after stumbling on one of the posts ;)

Want to practice that neat trick? Check out the EBGP Sessions over IPv6 LLA Interfaces lab exercise.

In some newer routers BGP would not be such a big bottleneck, but you need a lot of knob turning in BGP to get it right, while in LISP it is quite simple.

If you have many thousands concurrent airplanes with multi-link and max. 16 subnets with different routing policies on each, and the radio links are going up and down, then you have a large number of mobility events.

netlab allows you to define a VRF in the lab topology vrfs dictionary (global VRF) or in a node vrfs dictionary (node VRF). In most cases, you’d define a few global VRFs and move on.

Absolutely right. I’m working at an MSP, and we do a lot of project work for enterprises with between 500 and 2000 people. That means the IT department is not that big; it’s usually just a cost center for them.

So far, so good. However, whenever there’s a change, there’s an opportunity for marketing FUD, coming from the usual direction.

You can download the presentation or watch it on YouTube. Enjoy ;)

I was arguing for VxLAN EVPN with some of my peers, but I had no direct hands-on knowledge of how it would actually perform and very limited ability to lab it on hardware. My client was considering deploying Campus VxLAN, and they have one of the largest campuses in North America.

TL&DR: Data protection requirements like PCI-DSS aren’t there to make companies more secure but to make it too expensive for them to hoard excessive customer data (see also: GDPR).

Let’s start with the background story. When we added FRRouting containers support to netlab, someone decided to use lo0 as the loopback interface name. That device doesn’t exist in a typical Linux container, but it’s not hard to add it:

$ ip link add lo0 type dummy
$ ip link set dev lo0 up

In my larger environments, I see significant neighbor table cache entries, especially on network segments with hosts that make many long-term connections. These hosts have 10 to 20 addresses that maintain state over days or weeks to accomplish their processes.

What’s going on? A perfect storm of numerous unrelated annoyances:

His list includes well-known entries like twitter.com, azure.com, and github.com plus a few unexpected ones. I find cloudflare.net not having an AAAA DNS record truly hilarious. Someone within the company that flawlessly provided my website with IPv6 connectivity for years obviously still has some reservations about their own dogfood ;)

Most of the HA clustering solutions for stateful firewalls that I know implement a single-brain model, where the entire cluster is seen by the outside network as a single node. The node that is currently primary runs the control plane (hence, I call it single-brain). Sessions and the forwarding plane are synchronized between the nodes.

I would like your advice or a reference to a security framework I must consider when building a green field backbone in SR/MPLS.

Before going into the details, keep in mind that the core SR/MPLS functionality is not much different than the traditional MPLS:

• OSPFv3 in VRFs, implemented on Arista EOS, Cisco IOS, Cisco IOS-XE, FRR, and Junos (vMX, vPTX, vSRX).
• EBGP sessions over IPv4 unnumbered and IPv6 LLA interfaces on Arista EOS
• Cisco IOS XRd container support
• Retry tests until the timeout functionality in netlab validate.

This time, most of the work was done behind the scenes¹.

Bonus point: pointer to RFC 602 written in December 1973.

It strikes me that the entire industry lost out when we didn’t do SPB or TRILL. Specifically, I like how Avaya did SPB.

Oh, we did TRILL. Three vendors did it in different proprietary ways, but I’m digressing.

That should be an easy task if you configured MPLS in the past, so try to spice it up a bit:

• Use SR/MPLS instead of LDP
• Do it on a platform you’re not familiar with (hint: Arista vEOS is a bit different from Cisco IOS)
• Try to get it running on FRR containers.

Our leaf-and-spine fabric will have four leaves and two spines (but feel free to adjust the lab topology fabric parameters to build larger fabrics). The fabric will provide layer-2 connectivity to orange and blue VLANs. Two hosts will be connected to each VLAN to check end-to-end connectivity.

wemulate acts as a bump in the wire; it uses Linux bridges to connect two container interfaces. We’ll use it to introduce jitter into an IP subnet:

┌──┐   ┌────────┐   ┌──┐
│h1├───┤wemulate├───┤h2│
└──┘   └────────┘   └──┘                       
◄──────────────────────►
     192.168.33.0/24    

Just remember (as someone from Cisco TAC told me in those days) that “you might be the only one in the world doing it and might hit bugs no one has seen before.”

• Traditional networking operating systems enter only the OSPF route into the IP routing table.
• FRRouting enters OSPF and IBGP routes into the IP routing table.
• On all platforms I’ve tested, only the OSPF route gets into the forwarding table¹.

One could conclude that it’s perfectly safe to advertise the same prefixes in OSPF and IBGP. The OSPF routes will be used within the autonomous system, and the IBGP routes will be propagated over EBGP to adjacent networks. Well, one would be surprised 🤦‍♂️

Interestingly, most routing protocols span the whole seven layers of the OSI stack, with some layers implemented internally and others offloaded to other standardized protocols.

• Specify the default device type
• Enable the fabric plugin
• Specify the number of leaves and spines in the fabric.

For example, the following lab topology builds a fabric with Arista cEOS containers having two spines and four leaves:

You can practice the default filters you should always deploy on EBGP sessions with your customers in the Stop the Propagation of Configuration Errors lab exercise.

• Routing protocols compute the best paths to all known prefixes.
• These paths compete for entry in the routing table. The path(s) with the lowest administrative distance win.
• The entries from the routing table are fully evaluated (in particular, their next hops) and entered in the forwarding table.

Let’s use a simple BGP+OSPF network to illustrate what I’m talking about:

To make matters worse, my Google-Fu failed me when I tried to find a decent step-by-step configuration guide; all I got was a 12-minute video full of YouTube ads. Let’s fix that.

TL&DR: Yes, you can do that.

Now for the recipe:

Urs published the video recording of the presentation on YouTube; hope you’ll like it, and if you don’t get too annoyed by the overly pushy ads, watch the other videos from his infrastructure-as-code course.

I actually have worked with a few orgs that mix vendors at both spine and leaf layer. Can’t take names but they run fairly large streaming services. To me it seems like a play to avoid vendor lock-in, drive price points down and be in front of supply chain issues.

As always, one has to keep two things in mind:

We also plan to peer with multiple external ISPs to advertise our public IP space not directly from our PE routers but from dedicated Internet Routers, adding a firewall between our PEs and external Internet routers.

They could either run BGP between the PE routers, firewall, and WAN routers (see BGP as High-Availability Protocol for more details) or run BGP across a bump-in-the-wire firewall:

The routers advertising a summarized prefix should be connected by a path going exclusively through the part of the network with more specific prefixes. GRE tunnel also satisfies that criteria; the proof is left as an exercise for the reader.

One of my readers asked for a lengthier explanation, so here we go. Imagine a network with two areas doing inter-area summarization on /24 boundary:

Milan Zapletal submitted the source code for a huge lab topology they built with netlab. It has almost 50 routers and over 50 Linux nodes to emulate end-users and servers.

They used netlab to configure VLANs, VRFs, IS-IS, OSPF, EIGRP, BGP, MPLS, VXLAN, and EVPN. Imagine how long it would take to configure all that by hand using a more traditional labbing tool.

Every single job description that requires some sort of certification must be treated with suspicion. Demanding a certification usually means that you don’t know what you want, and you’re just outsourcing your thinking to someone else.

Have fun!

His idea is one of the best uses of generative AI in networking I’ve seen so far, as long as you remember that you’re dealing with an overconfident intern who has no problem making up an answer just to sound smart. Have fun!

Finally, if you don’t want to use ChatGPT (I wouldn’t blame you) or send captured data into The Cloud, someone already adapted his idea to use local LLMs.

---
- name: Test playbook
  hosts: localhost
  tasks:
  - set_fact:
      a: >
        {{ 123 == 345 or
           123 > 345 }}

It works every time 50% of the time (this time depending on your Ansible version).

Not surprisingly, that was bound to upset a few people, and Roman Dodin quickly pointed out the EVPN interoperability tests:

I had a hard time finding reasonable container images for BIRD; the BIRD team does not publish them, and everything else I found looked either abandoned or a hobby project. The solution turned out to be exceedingly simple: you cannot run the containers without Docker anyway, which means the docker build command is just a few keystrokes away. I added Dockerfiles needed to build those containers to the netlab source code and implemented the netlab clab build command as a thin wrapper around docker build. It takes just a few seconds (plus the time it takes to download the Ubuntu container image) to build the containers you need.

Intent-based infrastructure-as-code digital twins lifecycle management system

The podcast was published a few days ago; listen to it on the PacketPushers website or YouTube.

I was first using VyOS, but it uses the ISC DHCP relay, and that software relays unicast packets. The DHCP procedures eventually worked fine, but getting sensible outputs and explanations was a nightmare.

I quickly reproduced the behavior, but it took me almost half a year to turn it into a blog post. Engaging in a round of yak shaving (I wanted to implement DHCP in netlab first) didn’t exactly help, either.

Now imagine you decide to use netlab. Out of the box, you get topology management, lab orchestration, IPAM, routing protocol design (OSPF, BGP, and IS-IS), and device configurations, including IP routing and VLANs.

When I created the initial device configuration deployment playbook in netlab, I wanted to:

• Be able to specify a list of modules to provision.¹
• Provision just the modules used in the topology and specified in the list of modules.

This allows you to use netlab initial to deploy all configuration modules used in a lab topology or netlab initial -m ospf to deploy just OSPF while surviving netlab initial -m foo (which would do nothing).

Side note: I keep collecting links to insightful Git articles in the Git and GitHub section of the Network Automation Tools webinar.

Let’s fix that. Most BGP implementations have some remove private AS functionality that scrubs AS paths during outgoing update processing. You can practice it in the Remove Private BGP AS Numbers from the AS Path lab exercise.

You might have deployed a change that works perfectly fine from a network perspective but broke a customer application (for example, due to undocumented usage), so you must be able to return to the previous state even if everything works. Everybody says you need to “roll forward” (improve your change so it works), but you don’t always have that luxury and might need to take a step back. So, change tracking is essential.

He’s right: the undo functionality we take for granted in consumer software (for example, Microsoft Word) has totally spoiled us.

Would you apply BGP route maps with a peer/policy template or directly to a BGP neighbor? Of course, it depends; however, I believe in using a template for neighbors with the same general parameters and being more specific per neighbor when it comes to route manipulation.

As my reader already pointed out, the correct answer is It Depends, now let’s dig into the details ;)

Interestingly, that’s not the case on transparent bridges. Even though they have multiple interfaces, the whole bridge has a single MAC address, so one could claim we’re addressing nodes connected to a single data link layer. The IEEE standard is unambiguous: in every relevant diagram, the MAC address sits on top of multiple interfaces because the MAC address belongs to the control plane.

Wouldn’t it be nice if you could group those parameters into a template and apply the template to a neighbor? Most BGP implementations have something along those lines. That feature could be called a session template or a peer group, and you can practice it in the next BGP lab exercise.

Too often in the past, an overly ingenious engineer or programmer got the idea to simplify everyone’s life and use the data link layer addresses as the ultimate addresses of individual nodes. They would then put the transport layer on top of that to get reliable packet transport. Finally, put whatever application on top of the transport layer. Problem solved.

What Am I Talking About?

Imagine you have a bunch of IP prefixes you want to advertise with BGP. You could use network statements within the router bgp configuration to get that done:

That’s one of the many use cases for the local-as functionality available in most BGP implementations. You can practice it in the Use Multiple AS Numbers on the Same Router lab exercise.

Ansible has a complex system of variable (fact) precedence, which mostly makes sense considering the dozen places where a variable value might be specified (or overwritten). Ansible documentation also clearly states that the extra variables (specified on the command line with the -e keyword) have the highest precedence.

Now consider these simple playbooks. In the first one, we’ll set a fact (variable) and then print it out:

Not surprisingly, there’s a nerd knob for that (AS override), and you can practice it in the next BGP lab exercise: Fix AS-Path in Environments Reusing BGP AS Numbers.

How could I use NSX to create a cloud-like software network layer enabling a VMware enterprise to create a public cloud-like availability zone concept within a data center (something like Oracle Cloud does)?

That’s easy: stop believing in VMware marketing shenanigans.

We discussed those details in a lively Packet Pushers podcast with Claudia de Luna, David Sinn, Dinesh Dutt, Drew Conry-Murray and Ethan Banks. Have fun!

• Interfaces are created through standard Linux tools;
• You have to start the FRR management CLI from the Linux shell;
• If you need a routing daemon (for example, the BGP daemon), you must enable it in the FRR configuration file and restart FRR.

A new lab exercise covers these intricate details and will help you get fluent in configuring BGP on FRR or Cumulus Linux virtual machines or containers.

They measured the failover times caused by the primary uplink loss and figured out it takes more than five minutes to reestablish Internet connectivity to their site.

What do you get when you write code next to a Christmas tree? You can expect to get tons of eye candy, and that’s what netlab release 1.7.1 is all about.

It all started with a cleanup idea: I could replace the internal ASCII table-drawing code with the prettytable library. Stefan was quick to point out that I should be looking at the rich library, and the rest is history:

You might like the workaround I had to implement to keep those PDFs accessible: they are no longer behind a regwall.

You can find the list of all the free content ipSpace.net content here. The Conferences and Presentations page is another source of links to public presentations.

Unlike IGP protocols, BGP is not dependent on a single type of metric to choose the best path.

EIGRP is an immediate counterexample that brought the above quote to my attention, but it’s worth exploring the topic in more detail.

I tried to make that process a bit more structured and create external storage for my lab ideas – I started publishing more details on future BGP lab scenarios. The lab descriptions contain a high-level overview of the challenge and the lab topology; the details will be filled in later.

Want to know what’s coming in 2024? Check out the Upcoming Labs page of the BGP Labs project.

I am inspired to learn AWS advanced networking concepts and came across your website and webinar resources. But I cannot access it.

That is not exactly true. I wrote more than 4000 blog posts in the past, and some of them dealt with public cloud networking. There are also the free videos, some of them addressing public cloud networking.

One annoyance is what IP address gets used by default by the system for outbound traffic. It would be nice to have a generic OS-level way to say, “This IP on lo0 should be default for outbound IP traffic unless to the connected link subnet itself.”

That’s definitely a tough nut to crack, and Cathal described a few solutions he used in the past:

Hopefully, you won’t have to deal with something similar in real life, but then we know that crazy requirements trump good designs any day of the week.

How about using the collective efforts of the team developing device configuration templates for netlab? As of December 2023 netlab supports:

The easier it is to train an AI to do something, the less economically valuable that thing is. After all, the huge supply of the thing is how the AI got so good in the first place.

TL&DR: AI can easily disrupt things that are easy to generate and thus have little value. Seeing investors trying to recoup the billions pouring into the latest fad will be fun.

Hint: you give an AI model 32 step-by-step examples before asking a question, and it still gets it wrong 10% of the time.

Like any complex enough tool, netlab eventually had to deal with inconsistent version-specific functionality and configuration syntax (OK, topology attributes). I stumbled upon this challenge when I wanted to make labs that use two types of configurable devices.

In a recent blog post, Daniel Dib described a fantastic scenario: using a simple “why can’t I connect to a web site” question, explore everything from ARP/ND to DNS and TLS.

Obviously, you’ll never see anything that sane in a certification test. An interactive interview doesn’t scale (beyond CCDE), and using humans (and common sense judgment) creates potential legal liabilities (there were rumors that had been one of the reasons a talk with a proctor who could flunk you was dropped from the CCIE test).

Most companies want vendor-supported tools that will actually help them be more efficient, reduce human error, and increase the velocity at which the network team can support new apps and services.

Yeah, that’s nothing new. Most Service Providers wanted vendors to add tons of nerd knobs to their products to adapt them to existing network designs. Obviously, it must be done for free because a vast purchase order¹ is dangling in the air. We’ve seen how well that worked, yet learned nothing from that experience.

Want to use netlab? Jeroen van Bemmel implemented baseline SRv6 support for Nokia SR OS.

For an overview of netlab IPAM, watch the netlab address management video (part of the Network Automation Tools webinar), for more details read the netlab addressing tutorial.

AMS-IX published a very sanitized and diplomatic post-mortem incident summary in which they explained the outage was caused by LACP leakage. That phrase should be a red flag, but let’s dig deeper into the details.

• You can run automated lab validation tests with the netlab validate command. I will explain how I use that in BGP labs in a few days.
• If you want to build large leaf-and-spine topologies, you’ll love the fabric plugin.
• The bgp.domain plugin allows you to create topologies with multiple sites using the same BGP AS number.
• The bgp.policy plugin got AS-path prepending.
• bgp.originate plugin can be used to originate BGP IPv4 and IPv6 prefixes.