Your browser failed to load CSS style sheets. Your browser or web proxy might not support elliptic-curve TLS

Building network automation solutions

6 week online course

Start now!

Worth Reading: Magical Thinking in Internet Security

Someone pointed me to this article by dr. Paul Vixie (of the DNS fame). The best part (as I’m not a security person):

The TCO of new technology products and services, including security-related products and services, should be fudge-factored by at least 3X to account for the cost of reduced understanding. That extra 2X is a source of new spending: on training, on auditing, on staff growth and retention, on in-house integration.

In case you didn’t get it: figure out how much you think the magic unicorn-based software-defined solution will cost, then multiply it by three. Of course nobody wants to admit that.

see 1 comments

Video: Automated Data Center Fabric Deployment Demo

I was focused on network automation this week, starting with a 2-day workshop and continuing with an overview of real-life automation wins. Let’s end the week with another automation story: automated data center fabric deployment demonstrated by Dinesh Dutt during his part of Network Automation Use Cases webinar.

You’ll need at least free subscription to watch the video.

Add comment

Speakers in the Spring 2018 Building Next-Generation Data Center Online Course

We managed to get another awesome lineup of speakers for the Spring 2018 Building Next-Generation Data Center online course.

Russ White, one of the authors of CCDE and CCAr programs and highly respected book author will start the course with a topic everyone should always consider when designing new infrastructure: how do you identify tradeoffs and manage complexity, making sure you meet the customer requirements while at the same time having an easy-to-operate infrastructure.

read more Add comment

I Can’t Choose the Gear for You

One of my readers sent me a question along these lines after reading the anti-automation blog post:

Your blog post has me worried as we're currently reviewing offers for NGFW solution... I understand the need to keep the lid on the details rather than name and shame, but is it possible to get the details off the record?

I always believed in giving my readers enough information to solve their challenges on their own (you know, the Teach a man to fish idea).

read more Add comment

Streaming Telemetry Standards: So Many to Choose From

Continuing the Streaming Telemetry saga, let’s focus on presentation formats and transport mechanisms.

I already mentioned three presentation formats: XML (used by NETCONF), JSON (used by RESTCONF) and Protocol Buffers (used by gRPC). Two of them are text-based, the third one (Protocol Buffers) is binary encoding not unlike ASN.1 BER used by SNMP. That can’t be good in a JSON-hyped world, right?

read more see 3 comments

Should You Build or Buy an Automation Solution?

One of the most important aspects of the introductory part of my Building Network Automation Solutions online course is the question should I buy a solution or build my own?

I already described the arguments against buying a reassuringly-expensive single-blob-of-complexity solution from a $vendor, but what about using point tools?

read more see 3 comments

Worth Reading: How to Talk to a C-Level Executive

Ever wondered who manages to produce deja-moo like this one and why they’d do it?

We unveiled a vision to create an intuitive system that anticipates actions, stops security threats in their tracks, and continues to evolve and learn. It will help businesses to unlock new opportunities and solve previously unsolvable challenges in an era of increasing connectivity and distributed technology.

As Erik Dietrich explains in his blog post, it’s usually nothing more than a lame attempt to pretend there are some clothes hanging on the emperor.

Just in case you’re interested: we discussed the state of Intent-Based Majesty’s wardrobe in Network Automation Use Cases webinar.

Add comment

Linux Interfaces on Software Gone Wild

Continuing the Linux networking discussion we had in Episode 86, we focused on Linux interfaces in Episode 87 of Software Gone Wild with Roopa Prabhu and David Ahern.

We started with simple questions like “what is an interface” and “how do they get such weird names in some Linux distributions” which quickly turned into a complex discussion about kernel objects and udev, and details of implementing logical interfaces that are associated with ASIC front-panel physical ports.

read more Add comment

Before Commenting on Someone Mentioning RFC1925 ;)

Some of my readers got annoyed when I mentioned Google’s BeyondCorp and RFC 1925 in the same sentence (to be perfectly clear, I had Rule#11 in mind). I totally understand that sentiment – reading the reactions from industry press it seems to be the best thing that happened to Enterprise IT in decades.

Let me explain in simple terms why I think it’s not such a big deal and definitely not something new, let alone revolutionary.

read more see 2 comments

Who’s Pushing Layer-2 VPN Services?

Here’s another great point Tiziano Tofoni raised in his comment to my EVPN in small data center fabrics blog post:

I cannot understand the usefulness of L2 services. I think that the preference for L2 services has its origin in the enterprise world (pushed by well known $vendors) while ISPs tend to work at Layer 3 (L3) only, even if they are urged to offer L2 services by their customers.

Some (but not all) ISPs are really good at offering IP transport services with fixed endpoints. Some Service Providers are good at offering per-tenant IP routing services required by MPLS/VPN, but unfortunately many of them simply don’t have the skills needed to integrate with enterprise routing environments.

read more see 9 comments

Model-Driven Telemetry Isn’t as New as Some People Think

During the Campus Evolution with Cat9K presentation (I hope I got it right - the whole event was an absolute overload) the presenter mentioned the benefits of brand-new model-driven telemetry, which immediately caused me to put my academic hat on and state that we had model-driven telemetry for at least 30 years.

Don’t believe me? Have you ever looked at an SNMP MIB description? Did it look like random prose to you or did it seem to have some internal structure?

read more see 7 comments

Not Interested in Network Automation? No Problem (for now)

In the Business Impact of Network Automation podcast Ethan Banks asked an interesting question: “what will happen with older networking engineers who are not willing to embrace automation

The response somewhat surprised me: Alejandro Salisas said something along the lines “they’ll be just fine” (for a while).

Let me recap his argument and add a few twists of my own:

read more see 1 comments

Worth Reading: There Are No Enterprises and Service Providers

Russ White wrote a great article along the lines of what we discussed a while ago. My favorite part:

There are companies who consider the network an asset, and companies that consider the network a necessary evil.


On a tangential topic: Russ will talk about network complexity in the Building Next-Generation Data Center online course starting on April 25th.

Add comment

Video: Create an NSX Logical Switch with PowerNSX

After introducing PowerNSX Anthony Burke illustrated how easy it is to use with a Hello, World equivalent: creating a logical switch (VXLAN segment).

You’ll need at least free subscription to watch the video.

Want to know more about VMware NSX? We’ll run an NSX-focused event and a NSX Deep Dive workshop in Zurich on April 19th 2018, an overview webinar comparing NSX, ACI and EVPN on March 1st, and a deep dive in VMware NSX architecture later in 2018.

Add comment

Lack of Fast Convergence in SD-WAN Products

One of my readers sent me this question:

I'm in the process of researching SD-WAN solutions and have hit upon what I believe is a consistent deficiency across most of the current SD-WAN/SDx offerings. The standard "best practice" seems to be 60/180 BGP timers between the SD-WAN hub and the network core or WAN edge.

Needless to say, he wasn’t able to find BFD in these products either.

Does that matter? My reader thinks it does:

read more see 23 comments

Single-Image Systems or Automated Fabrics

In the Network Automation 101 webinar and Building Network Automation Solutions online course I described one of the biggest challenges the networking engineers are facing today: moving from thinking about boxes and configuring individual devices to thinking about infrastructure and services, and changing data models which result in changed device configurations.

The $1B question is obviously: and how do we get from here to there?

read more see 8 comments

Upcoming Events

In March 2018, we’ll continue the crazy content producing pace you’ve seen in January and February:
  • We’ll have the first part of NSX, ACI or EVPN webinar on March 1st. This session will cover the basics (don’t expect too many details), a follow-up session on April 24th with Mitja Robas will go into design considerations;
  • The EVPN Technical Deep Dive series with Dinesh Dutt starts on March 6th;
  • Elisa and Paolo will run the final part of Network Visibility with Flow Data on March 8th;
  • Last webinar in March: another installment in the leaf-and-spine saga – Multi-Pod and Multi-Site Fabrics with Lukas Krattiger on March 29th;
March is also the Troopers month. I’ll run a Hands-On Network Automation workshop there and have a motivational presentation during the main conference.
read more Add comment

Anti-Automation from the Antimatter Universe

One of my readers sent me a vivid description of his interactions with one of the so-called next-generation firewall vendors. Enjoy!

We’re using their highly promoted Next Generation Firewall (NGFW) management solution. New cutting edge software, centralized manager… but no CLI for configuration (besides some initial bootstrap commands). "You don't need that because everything is managed from our centralized manager GUI", says $vendor sales managers.

read more see 13 comments

EVPN with MPLS Data Plane in Data Centers

Mr. Anonymous (my most loyal reader and commentator) sent me this question as a comment to one of my blog posts:

Is there any use case of running EVPN (or PBB EVPN) in DC with MPLS Data Plane, most vendors seems to be only implementing NVO to my understanding.

Sure there is: you already have MPLS control plane and want to leverage the investment.

read more see 8 comments

Big Red Button for Network Automation

A while ago I was enjoying a few beers with a longtime friend of mine who happens to be running the networking team for one of the rare companies that understands how infrastructure should be built and operated.

Of course, I had to ask him what he thinks about the imminent death of CLI and all-encompassing automatic provisioning from some central orchestration system. Here’s the gist of his response:

read more see 1 comments

How Useful Is Microsegmentation?

Got an interesting microsegmentation-focused email from one of my readers. He started with:

Since every SDDC vendor is bragging about need for microsegmentation in order to protect East West traffic and how their specific products are better compared to competition, I’d like to ask your opinion on a few quick questions.

First one: does it even make sense?

read more see 9 comments

Automation Isn’t About Building a Button to Press

This is a guest blog post by Carl Buchmann, Managing Solution Consultant at TeraMach. Carl attended the Building Network Automation Solutions online course in 2017.

There is one thing I regret not doing sooner during my automation journey, and that is adopting Git and a proper IDE/text editor that has built-in source control management. I personally use Microsoft Visual Studio Code, as it has Git built in and has many great extensions to validate code syntax.

read more Add comment

Worth Reading: Whiteboxes for Everyone

Gian Paolo Boarina wrote a blog post describing why it’s so ridiculous to see everyone excited about the latest thing Netflix (or Google or Amazon or…) managed to pull off. Absolutely worth reading.

On a similar topic: did you notice that Google started promoting clientless SSL VPN as the next great thing? RFC 1925 anyone?

see 4 comments

[Video] Configure Data Center Devices with PowerShell

PowerShell started as a tool to automate Windows servers. It was picked up by VMware (and others) as a platform on which they built their own solutions (PowerCLI and PowerNSX)… but did you know you can use it to configure data center infrastructure, including NX-OS switches, SAN networks, and Cisco UCS?

In the Configuring Data Center Devices with PowerShell video, Mitja Robas described how to do that, and provided source code for all his examples.

You’ll need at least free subscription to watch the video.

Add comment

How Self-Sufficient Do You Want to Be?

The first car I got decades ago was a simple mechanical beast – you’d push something, and a cable would make sure something else moved somewhere. I could also fix 80% of the problems, and people who were willing to change spark plugs and similar stuff could get to 90+%.

Today the cars are distributed computer systems that nobody can fix once they get a quirk that is not discoverable with level-1 diagnostic tools.

read more see 3 comments

ExpertExpress Evolved into a Team of Experts

Years ago, I decided to try out another idea: solving real-life challenges with the help of an easy-to-consume online consulting service. When I discussed the idea with my friends during one of the early Networking Field Day events the opinion was pretty unanimous: “this will never work”

Fortunately, they were wrong. Not only did ~100 customers decided to use it in the meantime, the simple idea grew to a point where I couldn’t do it all on my own.

read more Add comment

EVPN Is More than VPLS on Steroids

Tiziano Tofoni wrote a lengthy comment on my EVPN in small data center fabrics blog post continuing the excellent discussion we started over a beer last October. Today I’ll address the first part:

I think that EVPN is an excellent standard for those who love Layer 2 (L2) services, we may say that it is an evolution of the implementation of the VPLS service, which addresses some limits in the original standard (RFCs 4761 and 4762).

I might be missing something, but in my opinion there’s no similarity between EVPN and VPLS (apart from the fact that they’re trying to solve the same problem).

read more see 1 comments

Automation Win: MPLS/VPN Service Deployment

I always encourage the students attending the Building Network Automation Solutions online course to create solutions for problems they’re facing in their networks instead of wasting time with vanilla hands-on assignments.

Francois Herbet took the advice literally and decided to create a solution that would configure PE-routers and create full-blown device configurations for CE-routers.

read more see 3 comments

Worth Reading: Networking Really Runs on Rainbows

From the fantastic Lines, Radios and Cables (a MUST READ if you’re even remotely interested in this thing called latency):

When we put different colours of light, or wavelengths, onto a single fibre, we call it Wave Division Multiplexing (WDM) which is a complicated way of saying a pretty rainbow […] International trading is powered by rainbows, literally.
Add comment

Video: What Is PowerNSX?

One of the beauties of VMware NSX is that it’s fully API-based – you can automate any aspect of it by writing a script (or using any of the network automation tools) that executes a series of well-defined (and well-documented) API calls.

To make that task even easier, VMware released PowerNSX, an open-source library of PowerShell commandlets that abstract the internal details of NSX API and give you an easy-to-use interface (assuming you use PowerShell as your automation tool).

read more see 1 comments

Want to Learn More about Docker and Containers?

One of my readers wanted to know more about containers and wondered how materials could help him. Here’s a short step-by-step guide:

I published this blog post to help subscribers navigate through Docker- and containers-related material. You might want to skip it if you’re not one of them.

read more Add comment

Automation Win: Cleanup Checkpoint Configuration

Gabriel Sulbaran decided to tackle a pretty challenging problem after watching my Ansible for Networking Engineers webinar: configuring older Checkpoint firewalls.

I had no idea what Ansible was when I started your webinar, and now I already did a really simple but helpful playbook to automate changing the timezone and adding and deleting admin users in a Checkpoint firewall using the command and raw modules. Had to use those modules because there are no official Checkpoint module for the version I'm working on (R77.30).

Did you automate something in your network? Let me know!

see 1 comments

Using EVPN in Very Small Data Center Fabrics

I had an interesting “how do you build a small fabric without throwing every technology in the mix” discussion with Nicola Modena and mentioned that I don’t see a reason to use EVPN in fabrics with just a few switches. He disagreed and gave me a few good scenarios where EVPN might be handy. Before discussing them let’s establish a baseline.

The Setup

Assume you’re building two small data center fabrics (small because you have only a few hundred VMs and two because redundancy and IT auditors).

read more see 6 comments

Machine Learning and Network Traffic Management

A while ago Russ White (answering a reader question) mentioned some areas where we might find machine learning useful in networking:

If we are talking about the overlay, or traffic engineering, or even quality of service, I think we will see a rising trend towards using machine learning in network environments to help solve those problems. I am not convinced machine learning can solve these problems, in the sense of leaving humans out of the loop, but humans could set the parameters up, let the neural network learn the flows, and then let the machine adjust things over time. I tend to think this kind of work will be pretty narrow for a long time to come.

Guess what: as fancy as it sounds, we don’t need machine learning to solve those problems.

read more Add comment

Brief Recap: Tech Field Day at Cisco Live Europe 2018

I don’t think I’ve ever been at a Tech Field Day event that’s been as intense as what we went through in the last few days at Cisco Live Europe – at least 17 different presentations in two days. It’s still all a blur and will take a long while to sort out.

First impressions:

read more see 7 comments

First Speakers in Building Next-Generation Data Center Online Course

Although it’s almost three months till the start of the Building Next-Generation Data Center online course, we already have most of the guest speakers. Today I’d like to introduce the first two (although they need no introduction).

You might have heard about Russ White. He was known as Mr. CCDE when that program started and recently focused more on data centers, open networking and whitebox switching. He’s also an authority on good network design and architecture, network complexity, and tradeoffs you have to make when designing a network.

read more Add comment

How to Become a Better Networking Engineer

Got an interesting set of questions from one of my readers. He started with:

I really like networks but I don't know if I am doing enough for this community. Most of my work is involved with technologies which are already discovered by people and I am not really satisfied with it.

Well, first you want to decide whether you want to be (primarily) a researcher (focusing on discovering new stuff), an engineer (mostly figuring out how to build useful things by using existing stuff), or an administrator (configuring stuff).

read more see 3 comments

Revisited: The Need for Stretched VLANs

Regardless of how much I write about (the ridiculousness of using) stretched VLANs, I keep getting questions along the same lines. This time it’s:

What type of applications require L2 Extension and L3 extension?

I don’t think I’ve seen anyone use L3 extension (after all, isn’t that what Internet is all about), so let’s focus on the first one.

Stretched VLANs (or L2 extensions) are used to solve a number of unrelated problems, because once a vendor sold you a hammer everything starts looking like a nail, and once you get used to replacing everything with nails, you want to use them in all possible environments, including public and hybrid clouds.

read more see 14 comments

Use YANG Data Models to Configure Network Device with Ansible

It took years after NETCONF RFCs were published before IETF standardized YANG. It took another half-decade before they could agree on how to enable or disable an interface, set interface description, or read interface counters. A few more years passed by, and finally some vendors implemented some of the IETF or OpenConfig YANG data models (with one notable exception).

Now that we have the standardized structure, it’s easy to build automated multi-vendor networks, right? Not so fast…

read more see 6 comments

Don't focus on trivia...

Found this great quote in Algorithms to Live By: The Computer Science of Human Decisions - a must-read for all nerds:

Depend upon it there comes a time when for every addition of knowledge you forget something that you knew before. It is of the highest importance, therefore, not to have useless facts elbowing out the useful ones.

Sherlock Holmes

Now you know why you should focus on how things work instead of memorizing commands ;)

Add comment

Video: Big- or Small-Buffer Switches

After describing the basics of internal data center switch architectures, JR Rivers focused on the crux of the problem the vendors copiously exploit to create a confusopoly: is it better to use big- or small-buffer switches?

You’ll need at least free subscription to watch the video.

see 3 comments

BGP in EVPN-Based Data Center Fabrics

EVPN is one of the major reasons we’re seeing BGP used in small- and mid-sized data center fabrics. In theory, EVPN is just a BGP address family and shouldn’t have an impact on your BGP design. In practice, suboptimal implementations might invalidate that assumption.

I've described a few EVPN-related BGP gotchas in BGP in EVPN-Based Data Center Fabrics, a section of Using BGP in Data Center Leaf-and-Spine Fabrics article.

Alex raised a number of valid points in his comments to this blog post. While they don't fundamentally change my view on the subject, they do warrant a more nuanced description. Expect an updated version of this part of the article when I return from Cisco Live Europe

see 19 comments

Hard Truths Not Taught in Schools

J Metz published a great article describing six hard truths not taught in school. As all good things should come in 7-tuples, here’s another one I was told ages ago when I was a young hotshot full of myself:

Professions were created for a reason – they enable people to do the work they’re qualified to do.

Needless to say, it took me decades to fully understand its implications.

read more see 2 comments

Synchronize Network Management Parameters across Network Devices

While I have stock homework assignments prepared for every module of the Building Network Automation Solutions online course I always encourage the students to pick a challenge from their production network and solve it during the course.

Pavel Rovnov decided to focus on consistency of network management parameters (NTP, SNMP, SSH and syslog configuration) across Extreme and Cumulus switches, Fortinet firewalls and several distributions of Linux.

read more Add comment

Packet Forwarding on Linux on Software Gone Wild

Linux operating system is used as the foundation for numerous network operating systems including Arista EOS and Cumulus Linux. It provides most networking constructs we grew familiar with including interfaces, VLANs, routing tables, VRFs and contexts, but they behave slightly differently from what we’re used to.

In Software Gone Wild Episode 86 Roopa Prabhu and David Ahern explained the fundamentals of packet forwarding on Linux, and the differences between Linux and more traditional network operating systems.

read more Add comment

Webinars in 2017

2017 was one of the busiest years since I started the project.

It started with an Ansible for Networking Engineers session covering advanced Ansible topics and network device configurations. Further sessions of that same webinar throughout 2017 added roles, includes, extending Ansible with dynamic inventory, custom modules and filters, and using NAPALM with Ansible.

read more Add comment

Ansible, Chef, Puppet or Salt? Which One Should I Use?

One of the first things I did when I started my deep-dive into network automation topics was to figure what tools people use to automate stuff and (on a pretty high level) what each one of these tools do.

You often hear about Ansible, Chef and Puppet when talking about network automation tools, with Salt becoming more popular, and CFEngine being occasionally mentioned. However, most network automation engineers prefer Ansible. Here are a few reasons.

read more see 4 comments

Event-Driven Automation on Building Network Automation Solutions Online Course

Most engineers talking about network automation focus on configuration management: keeping track of configuration changes, generating device configurations from data models and templates, and deploying configuration changes.

There’s another extremely important aspect of network automation that’s oft forgotten: automatic response to internal or external events. You could wait for self-driving networks to see it implemented, or learn how to do it yourself.

read more Add comment

Meltdown and Its Networking Equivalents

One of my readers sent me this question:

Do you have any thoughts on this meltdown HPTI thing? How does a hardware issue/feature become a software vulnerability? Hasn't there always been an appropriate level of separation between kernel and user space?

There’s always been privilege-level separation between kernel and user space, but not the address space separation - kernel has been permanently mapped into the high-end addresses of user space (but not visible from the user-space code on systems that had decent virtual memory management hardware) since the days of OS/360, CP/M and VAX/VMS (RSX-11M was an exception since it ran on 16-bit CPU architecture and its designers wanted to support programs up to 64K byte in size).

read more see 3 comments

Worth Reading: Robust IPAM

Elisa Jasinska covered several IPAMs in her overview of open-source network automation tools, and we had Jeremy Stretch talking about NetBox in the Building Network Automation Solutions online course, but if you’re looking for a really robust easy-to-implement solution, check out this document from 1998 (deployment experience, including a large-scale one).

see 3 comments

Upcoming Events

2018 has barely started and we’re already crazily busy:

The last week of January is Cisco Live Europe week. I’ll be there as part of the Tech Field Day Extra event – drop by or send me an email if you’ll be in Barcelona during that week.

read more see 1 comments

Fat Fingers Strike Again…

Level3 had a pretty bad bad-hair-day just a day before Pete Lumbis talked about Continuous Integration on the Building Network Automation Solutions online course (yes, it was a great lead-in for Pete).

According to messages circulating on mailing lists it was all caused by a fumbled configuration attempt. My wild guess: someone deleting the wrong route map, causing routes that should have been tagged with no-export escape into the wider Internet.

read more Add comment

BGP Route Selection: a Failure of Intent-Based Networking

It’s interesting how the same pundits who loudly complain about the complexities of BGP (and how it will be dead any time soon and replaced by an SDN miracle) also praise the beauties of intent-based networking… without realizing that the hated BGP route selection process represents one of the first failures of intent-based approach to networking.

Let’s start with some definitions. There are two ways to get a job done by someone else:

read more see 11 comments

New Design on

One of my readers sent me a polite email a while ago saying “your site is becoming like $majorVendor’s web site – every corner looks completely different based on when you made it

The worst part is that he was right, so I spent the last two weeks as a website janitor, mopping up broken markup, fixing CSS cracks, polishing old texts…

read more Add comment