TL&DR: You automate the whole process. What else do you expect?

During the Tech Field Day Extra @ Cisco Live Europe 2019 we were taken on a behind-the-stage tour that included a chat with people who built the Cisco Live network, and of course I had to ask how they automated the whole thing. They said “well, we have the guy that wrote the whole system onsite and he’ll be able to tell you more”. Turns out the guy was my good friend Andrew Yourtchenko who graciously showed the system they built and explained the behind-the-scenes details.

Trick#1: Keep it as simple as possible.

They didn’t use a complex general-purpose tool that could do everything and everything else and make you coffee (according to PowerPoints and whitepapers) but built a system that does exactly what they need and nothing more.

As always, it all starts with good network design – simple layer-3 core + access network (wireless L2 mobility is handled by wireless controllers)… and here’s the next neat detail: they use DHCP to assign IP address to access switch uplinks and to point the switches to initial configuration which can be the same for all switches because IP addresses are assigned dynamically.

Trick#2: Use EEM applet on the switch to collect its MAC address and serial number and report it to a web server.

If the switch is already known it’s provisioned, if it’s a new device a script logs into the switch, collects all the information it needs, and adds the new device into the staging area of device inventory where an operator takes over and configures it.

Trick#3: All changes to the network devices are done in the same way:

• Copy configuration changes to device flash.
• Execute copy flash:filename running-config

Dangerous changes are bracketed with reload in 15 and reload cancel so they don’t lose a switch if (for example) they have to change routing setup or uplink configuration.

All interactions with devices are logged and the printouts are saved, so you can always check how a particular change worked out (and why it might have failed).

Trick #4: Don’t be too smart dealing with configuration changes.

Most of the tools out there try to figure out how to get from here (current configuration) to there (desired configuration) with a carefully choreographed sequence of configuration commands. They decided to make it simpler than that:

• All interface configuration changes start with default interface so the interface is always in a known state. Keep in mind that they use copy to execute all configuration commands (almost) at once, resulting in somewhat-atomic operations (or at least as atomic as you can get with Cisco IOS).
• All other changes are stored as templates that have remove and add elements – it’s the responsibility of whoever is configuring the system to figure out what needs to be taken out and how to do it safely. Sounds like a really bad approach until you realize that you have to do that only when preparing a new change template… and in a well-designed network you don’t change your mind too often anyway.

… and there were probably a dozen other things Andrew mentioned and I forgot to write down.

Lesson learned: provisioning a large campus network in a few days is easy and reliable as long as you do your homework and keep things as simple as possible.

Want to do something similar but don’t know where to start? Check out our network automation course.