Category: automation

Some network automation skeptics came to that place the hard way: they got burned by half-baked semi-tested systems. This is what one of my good friends had to say in a LinkedIn comment:

I am suspicious of automation, as I’ve unfortunately seen too many outages caused by either human error or faulty automation. Every time it required human CLI/GUI intervention to correct it. The problem is that the more automation we push, the fewer people know how to use the “old school” way to administer stuff.

Network automation is not the only IT discipline that could cause hard-to-correct errors requiring manual intervention. I’m positive everyone knows at least one horror story resulting in manual tweaking of the Windows registry, or a sequence of arcane SQL commands¹.

Here’s some common-sense view on hard-coded rules versus machine learning in network operations by Mark Seery – quite often we can specify our response to an event as a simple set of rules, but if we want to identify deviation from “normal” behavior, machine learning might not be a bad idea.

For more details, watch the Event-Driven Network Automation part of Building Network Automation Solutions online course.

I keep getting questions along the lines of “is network automation practical/a reality?” with arguments like:

Many do not see a value and are OK with just a configuration manager such as Arista CVP (CloudVision Portal) and Cisco DNA.

Configuration consistently is a huge win regardless of how you implement it (it’s perfectly fine if the tools your vendor providers work for you). It prevents opportunistic consistency, as Antti Ristimäki succinctly explained:

Antti Ristimäki left an interesting comment on Network Automation Considered Harmful blog post detailing why it’s suboptimal to run manually-configured modern service provider network.

I really don’t see how a network any larger and more complex than a small and simple enterprise or campus network can be developed and engineered in a consistent manner without full automation. At least routing intensive networks might have very complex configurations related to e.g. routing policies and it would be next to impossible to configure them manually, at least without errors and in a consistent way.

I could spend days writing riffs on some of the more creative (in whatever dimension) comments left on my blog post or LinkedIn¹. Here’s one about uselessness of network automation in cloud infrastructure (take that, AWS!):

If the problem is well known you can apply rules to it (automation). The problem with networking is that it results in a huge number of cases that are not known in advance. And I don’t mean only the stuff you add/remove to fix operational problems. A friend in one of the biggest private clouds was saying that more than 50% of transport services are customized (a static route here, a PBR there etc) or require customization during their lifecycle (e.g. add/remove a knob). Telcos are “worse” and for good reasons.

Yeah, I’ve seen such environments. I had discussions with a wide plethora of people building private and public (telco) clouds, and summarized the few things I learned (not many of them good) in Address the Business Challenges First part of the Business Aspects of Networking Technologies webinar.

A few months ago, Urs Baumann created NetTowel, a very nice CLI wrapper around several popular libraries, including Jinja2, TTP, NetMiko and netaddr. Although it seems he got busy with other things in recent months, and the development stalled a bit, the tool is definitely worth exploring.

Some of the blog comments never cease to amaze me. Here’s one questioning the value of network automation:

I think there is a more fundamental reason than the (in my opinion simplistic) lack of skills argument. As someone mentioned on twitter

“Rules make it harder to enact change. Automation is essentially a set of rules.”

We underestimated the fact that infrastructure is a value differentiator for many and that customization and rapid change don’t go hand in hand with automation.

Whenever someone starts using MBA-speak like value differentiator in a technical arguments, I get an acute allergic reaction, but maybe he’s right.

Responding to my Infrastructure as Code Sounds Scary blog post, Deepak Arora posted an interesting (and unfortunately way too accurate) list of challenges you might encounter when trying to introduce network automation in an enterprise environment.

He graciously allowed me to repost his thoughts on my blog.

Why don’t we agree on that :

One of my readers preparing for public cloud deployment sent me an interesting observation:

I pushed to use infrastructure-as-code as we move to Azure, but I’m receiving a lot of pushback due to most of the involved parties not having any experience with code. Management is scared to use any kind of “homegrown” tools that only a few would understand. I feel like I’m stuck deploying and managing the environment manually.

It looks like a bad case of suboptimal terminology for this particular audience. For whatever reason, some infrastructure engineers prefer to stay as far away from programming as possible¹, and infrastructure-as-code sounds like programming to them.

Andrea Dainese released an interesting tool that performs automated network discovery, pushes the discovered data into NetBox, and then uses netbox-topology-views plugin to create network topology diagrams.

Definitely worth exploring!

Using Terraform to deploy networking elements with an SDN controller that cannot replace the current state of a tenant with the desired state specified in a text file (because nobody ever wants to do that, right) sounds like a great idea… until you try to do it at scale.

Noël Boulene hit interesting scalability limits when trying to provision VLANs on Cisco ACI with Terraform. If you’re thinking about doing something similar, you REALLY SHOULD read his article.

Are you afraid the network automation will eat your job? You might have to worry if you’re a VLAN-provisioning CLI jockey, but then you’re not alone. Textile workers faces the same challenges in 19th century and automation report from 1958 the clerical workers were facing the same dilemma when the first computers were introduced.

Guess what: unemployment rate has been going up and down in the meantime (US data), but mostly due to various crisis. Automation had little impact.

I’m usually telling networking engineers seriously considering whether to automate their networks to cleanup their design and simplify the network services first.

The only reasonable way forward is to simplify your processes – get rid of all corner cases, all special deals that are probably costing you more than you earned on them, all one-off kludges to support badly-designed applications – and once you get that done, you might realize you don’t need a magic platform anymore, because you can run your simpler network using traditional tools.

While seasoned automation practitioners agree with me, a lot of enterprise engineers face a different reality. Straight from a source that wished to remain anonymous…

A friend of mine sent me the following question a few months ago:

I thought you might know the best way (currently) to create a digital clone of parts of a production network? The objective is to test changes against a test network as part of a CI/CD process. Ideally, there would be an automation that could replicate selected parts of a production network in a test network.

TL&DR: Sounds great, but you might be solving the wrong problem.

Lívio Zanol Puppim published a series of blog posts describing a full-stack network automation, including GitOps with GitLab, handling secrets with Hashicorp Vault, using Ansible and AWX to run automation scripts, continuous integration with Gitlab CI Runner, and topped it off with a REST API and React-based user interface.

You might not want to use the exact same components, but it’s probably worthwhile going through his solution and explore the source code. He’s also looking for any comments or feedback you might have on how to improve what he did.