Your browser failed to load CSS style sheets. Your browser or web proxy might not support elliptic-curve TLS

Building network automation solutions

6 week online course

Start now!

Packet Forwarding on Linux on Software Gone Wild

Linux operating system is used as the foundation for numerous network operating systems including Arista EOS and Cumulus Linux. It provides most networking constructs we grew familiar with including interfaces, VLANs, routing tables, VRFs and contexts, but they behave slightly differently from what we’re used to.

In Software Gone Wild Episode 86 Roopa Prabhu and David Ahern explained the fundamentals of packet forwarding on Linux, and the differences between Linux and more traditional network operating systems.

read more Add comment

Webinars in 2017

2017 was one of the busiest years since I started the project.

It started with an Ansible for Networking Engineers session covering advanced Ansible topics and network device configurations. Further sessions of that same webinar throughout 2017 added roles, includes, extending Ansible with dynamic inventory, custom modules and filters, and using NAPALM with Ansible.

read more Add comment

Ansible, Chef, Puppet or Salt? Which One Should I Use?

One of the first things I did when I started my deep-dive into network automation topics was to figure what tools people use to automate stuff and (on a pretty high level) what each one of these tools do.

You often hear about Ansible, Chef and Puppet when talking about network automation tools, with Salt becoming more popular, and CFEngine being occasionally mentioned. However, most network automation engineers prefer Ansible. Here are a few reasons.

read more Add comment

Event-Driven Automation on Building Network Automation Solutions Online Course

Most engineers talking about network automation focus on configuration management: keeping track of configuration changes, generating device configurations from data models and templates, and deploying configuration changes.

There’s another extremely important aspect of network automation that’s oft forgotten: automatic response to internal or external events. You could wait for self-driving networks to see it implemented, or learn how to do it yourself.

On March 20th live session of Building Network Automation Solutions online course David Gee will dive deeper into event-driven network automation. As he explains the challenge:

When it comes to running infrastructure and infrastructure services, a lot of the decision making is human based. Someone reads a ticket, someone decides what to do. Someone gets alerted to an event and that someone does something about it. This involvement causes friction in the smooth-running nature of automated processes. Fear not! Something can be done about it.

We all know the stories of ITIL and rigid process management and David will show you how event-driven automation could be made reality even with strict and rigid controls, resulting in an environment that reacts automatically to stimuli from your services and infrastructure. We will discuss what events are, when they're important, how to normalize them, and what we can do when we have identified an event positively. We will also discuss commercial vs open source options along with their pros and cons.

Finally, you will see a live demonstration of both syslog and ICMP powered event driven automation in action. Links to usable code samples will be provided in the session so you reproduce the demos in your own environment.

Interested? Register now!

Add comment

Meltdown and Its Networking Equivalents

One of my readers sent me this question:

Do you have any thoughts on this meltdown HPTI thing? How does a hardware issue/feature become a software vulnerability? Hasn't there always been an appropriate level of separation between kernel and user space?

There’s always been privilege-level separation between kernel and user space, but not the address space separation - kernel has been permanently mapped into the high-end addresses of user space (but not visible from the user-space code on systems that had decent virtual memory management hardware) since the days of OS/360, CP/M and VAX/VMS (RSX-11M was an exception since it ran on 16-bit CPU architecture and its designers wanted to support programs up to 64K byte in size).

read more see 3 comments

Worth Reading: Robust IPAM

Elisa Jasinska covered several IPAMs in her overview of open-source network automation tools, and we had Jeremy Stretch talking about NetBox in the Building Network Automation Solutions online course, but if you’re looking for a really robust easy-to-implement solution, check out this document from 1998 (deployment experience, including a large-scale one).

see 3 comments

Upcoming Events

2018 has barely started and we’re already crazily busy:

The last week of January is Cisco Live Europe week. I’ll be there as part of the Tech Field Day Extra event – drop by or send me an email if you’ll be in Barcelona during that week.

read more see 1 comments

Fat Fingers Strike Again…

Level3 had a pretty bad bad-hair-day just a day before Pete Lumbis talked about Continuous Integration on the Building Network Automation Solutions online course (yes, it was a great lead-in for Pete).

According to messages circulating on mailing lists it was all caused by a fumbled configuration attempt. My wild guess: someone deleting the wrong route map, causing routes that should have been tagged with no-export escape into the wider Internet.

read more Add comment

BGP Route Selection: a Failure of Intent-Based Networking

It’s interesting how the same pundits who loudly complain about the complexities of BGP (and how it will be dead any time soon and replaced by an SDN miracle) also praise the beauties of intent-based networking… without realizing that the hated BGP route selection process represents one of the first failures of intent-based approach to networking.

Let’s start with some definitions. There are two ways to get a job done by someone else:

read more see 11 comments

New Design on

One of my readers sent me a polite email a while ago saying “your site is becoming like $majorVendor’s web site – every corner looks completely different based on when you made it

The worst part is that he was right, so I spent the last two weeks as a website janitor, mopping up broken markup, fixing CSS cracks, polishing old texts…

read more Add comment

Unique IPv6 Prefix Per Host – How Complex Do You Want IPv6 to Be?

In December 2017 IETF published RFC 8273 created by the v6ops working group (which means there must have been significant consensus within the working group that we need the solution and that it makes at least marginal sense).

The RFC specifies a mechanism by which the first-hop router allocates a unique /64 IPv6 prefix for every host attached to a subnet and uses unicast and multicast RA responses sent to unicast MAC addresses to give every host the impression that it’s the sole host on its own subnet.

The first thought of anyone even vaguely familiar with how complex IPv6 already is should be “WTF???” Unfortunately, there are good reasons we need this monstrosity.

read more see 7 comments

Salt Deep Dive on Building Network Automation Solutions Online Course

In the first few sessions of the Building Network Automation Solutions online course we used Ansible as the tool-of-choice because it’s the easiest automation tool to get started with. Now that we’ve established the baseline, it’s time to explore the alternatives.

In a live session on February 27th 2018, Mircea Ulinic will describe Salt, an open source, general-purpose event-driven automation framework that we briefly discussed in Episode 77 of Software Gone Wild podcast.

read more Add comment

BGP: the Tragedy of the Commons

Every now and then someone looks at a few recent BGP incidents (from fat fingers to more dubious ones) and says “we need a better BGP”.

It’s like being unable to cope with your kids or your team members because you don’t have the guts to tell them NO and trying to solve the problem by implementing new procedures and rules.

Like anything designed on a few napkins BGP has its limit. They’re well known, and most of them have to do with trusting your neighbors instead of checking what they tell you.

The solutions to the problem are pretty simple and have been known for decades (BCP38 was published in May 2000). In a nutshell you have to:

  • Build a global repository of who owns what address space;
  • Document who connects to whom and what their peering policies are;
  • Filter the updates received from your customers and peers based on the information from those repositories;
  • Filter the traffic from addresses that are obviously spoofed.

We have most of the tools we need to get the job done; you’ll find them described in Best Current Practice (BCP) 194. It’s also not impossible to get the job done from the operational perspective. NTT has been doing it for quite a while; Job Snijders described their approach to practical BGP filtering in a NANOG67 presentation.

Unfortunately you’ll always find ISPs (including some so-called Tier-1 providers) who couldn’t care less about fixing things and making global Internet a better place, because implementing those rules might impact their sloppy customers, and it’s always easier to give in to your customer’s (or your kid’s) screaming instead of telling them “you can’t have the candy because you haven’t followed the rules”

The “only” problem of getting things done is that like in any dysfunctional family the kids (= customers) could go shopping around for someone more permissive, and they’ll always find another ISP with lower prices, more relaxed rules, and connectivity to a dysfunctional transit provider.

Even worse than individual sloppy ISPs – there are Internet Exchange Points running route servers with no filters. Job Snijders got so sick-and-tired of them that he added a public column-of-shame to his IXP overview spreadsheet. Not that it would help much; Geoff Huston has been producing deaggregation and excessive BGP updates reports for years with absolutely no visible effect.

Being good engineers who hate confrontations, we’re trying to sneak our way around those problems with various cryptographic tools (like RPKI) instead of fixing the source of the problem: chaotic (or non-existent) operational practices of some major players.

Unfortunately, you can never solve people- or process problems with new technology, you can just make them more convoluted and harder to troubleshoot. What we’d really need to have are driving licenses for ISPs, and some of them should be banned for good due to repetitive drunk driving. Alas, I don’t see that happening in my lifetime.

see 2 comments

Please Respond: Survey on Interconnection Agreements

Marco Canini is working on another IXP-related research project and would like to get your feedback on inter-AS interconnection agreements, or as he said in an email he sent me:

As academics, it would be extremely valuable for us to receive feedback from network operators in the industry.

It’s fantastic to see researchers who want to base their work on real-life experience (as opposed to ideas that result in great-looking YouTube videos but fail miserably when faced with reality), so if you’re working for an ISP please take a few minutes and fill out this survey.

Add comment