Do a Cleanup Before Automating Your Network
Remington Loose sent me an interesting email describing his views on the right approach to network automation after reading my Network Reliability Engineering Should Be More than Software or Automation rant – he’s advocating standardizing network services and cleaning up your network before trying to deploy full-scale automation.
I think you are 100% right to start with a thorough cleanup before automation. Garbage in, garbage out. It is also the case that all that inconsistency and differentiation makes for complexity in automation (as well as general operations) that makes it harder to gain traction.
Anecdotally, I see customers that have long-serving, intelligent, capable network engineer/architects have more stable, more consistent and more standardized networks. That’s a lot of caveats but it is the case. Customers without long-term, competent, senior folk have a potpourri of things and it detracts from stability as well as automation. As an outside advisor I try to slowly push customers into standardization as the first few steps…and it pays off. After a few years of working with folks I get a lot of compliments and comments about how stable and reliable their networks now are…and they are ready to make the next “moves” which include automation.
I don’t want to say people shouldn’t start automating their networks or that automation is secondary but rather that “automation is great, you should start the journey for sure, but perhaps first you should spend time cleaning first”. The analogy in my mind is like using a robot vacuum. They are great and can save you time. But you should pick up the room before turning it on. If not, you can get into a situation of vacuum-meets-poop-now-you-have-poop-everywhere or fragile things get broken or vacuum fails to provide value as it gets stuck/snagged. Then you have to clean-up the mess made by the vacuum, fix/clean the vacuum and still clean the floor…so your one problem just became three. Likewise, using automation before standardizing is using a tool to fix a broken process and that really doesn’t work in my experience.
I don’t think it is a grumpy older-timer thing but I do think newer engineers fail to full appreciation the value of standardization and simplicity in design. I am not sure if this is one of those lessons you have to learn for yourself but I suspect it is more of a lack of mentoring/apprenticeship in the networking field. It speaks to the value of your content (I think) for junior folk and it highlights the ongoing need for a different method of training network engineers.
Yes, standardization is the hard part and basically to happen to get the benefit of automation, but you can't expect an active shop to complete standardization before automation. You can definitely clean up a lot of cruft, but sometimes you need to chip away at the block.
In my current role, we learned a lot about the shape/size of services as we were automating them and adopting/converting bespoke eBGP services to standardized services. It took a lot of conversations with stakeholders (customers, CRM people, engineers, architects), lots of work extracting data from existing config, and some hard questions of "do you really need this, and why?" some of which we had to compromise on for time-sake.
In the end, I think we wound up with a fairly well designed service template and we made the right thing the easy thing, but also managed to make the necessary thing possible. We were lucky enough and had enough talent to do this during a hardware migration, so we got to chuck the config cruft out the window with
request system zeroize. Our change board was also very supportive of us getting stuff from dev into production relatively fast during that rollout.In a previous life, we worked really hard on standardization, but could never quite graduate past automation tasks because we constantly fought the second law of thermodynamics -- everything slowly decays into the inevitable heat-death of the universe. With the benefit of hindsight, I think we needed to automate the parts we had standardized and make those easy so we weren't fighting bad habits.
I agree with you, but I think some lip service needs to be paid to iteration. Getting part of something automated has a way of "locking in the automation" that can in and of itself be valuable.
I think what I'm trying to say can be boiled down to "Standardization needs to take place before or [more practically] as part of the automation journey, but you're never going to find yourself in a better place unless you standardize."
Absolutely. Many an engineer will have been or is planned to be made redundant by a management strategy due to impending automation…..when really they should be re-deployed to standardise first and be a part of the automation development.
I always remembered a quote on a Juniper blog that pretty much sums this up: “Rod Michael said, “If you automate a mess, you get an automated mess.” Automation must follow architecture and accuracy.”
https://blogs.juniper.net/en-us/enterprise-cloud-and-transformation/seven-deadly-deceptions-of-network-automation
I can’t find the origin of Rod Michaels “mess” quote but wise words.