Where Is my VLAN Provisioning Application?
Yesterday I wrote that it’s pretty easy to develop a VLAN provisioning application (integrating it with vCenter or System Center earns you bonus points, but even that’s not too hard), so based on the frequent “I hate using CLI to provision VLANs” rants you might wonder where all the startups developing those applications are. Simple answer: there’s no reasonably-sized market. How would I know that? We’ve been there.
When I was working at a system integrator we wrote numerous network configuration/provisioning applications, starting with provisioning of Internet customers on our dial-up servers and routers in 1993 to MPLS/VPN provisioning for a major MPLS/VPN-over-IPsec-over-IP-over-MPLS/VPN customer. Needless to say, we also had to fully automate our remote labs and cloud services (and both required frequent reconfiguration of numerous networking elements).
Could we productize those applications? Hardly. In most cases, we had to tailor the provisioning application to the actual hardware (we’ve learned the hard way how to abstract that part in our remote labs) and network topology – many networking architects and designers proudly create one unique network after another, each one of them a masterpiece of architecture and engineering, never considering the impact of their efforts on other elements in the food chain.
If you’ve ever been involved in an ERP system rollout, you probably know the tradeoffs you’d face when choosing a data center (or cloud) orchestration/provisioning platform. If you buy a mid-range ERP platform (Navision), you’re forced to adapt your business to its limitations, or you could choose a high-end platform (SAP) and be forever joined at the hip to an army of consultants constantly tweaking SAP to the nuances of your business.
The situation is no different in data center (or service provider or cloud) environment – you could easily find a framework platform that will cost you an arm and a leg (prices in 6-to-7 figure range) not including the customization and adaptation services, and large ISPs and cloud providers invested in developing their own solutions. You have to be big enough to justify spending that much money just to get a provisioning platform; most of us are not in that category.
On the other hand, it would be hard to find a SolarWinds of VLAN provisioning. Why? We’re not building data centers with standard architecture and components that would require minimum customization, and it’s close to impossible to write a low-priced application that will work well with millions of totally unique networks. I don’t understand why anyone expects SDN to change that, but please do feel free to write a comment if you disagree.
I also agree with Steven Iveson's comment that it should be possible to make manual management of your unique and beautiful snowflake network much easier by eliminating busywork and inconsistent configurations caused by human error (see Meraki for one example).
On the other hand, SDN is similar to cloud computing. It’s not a totally new technology it’s a necessary improvement that meets the current requirements of the IT departments. Networking equipment getting some kind of programmable interfaces for optimizing the provisioning process, improve the configuration management, etc. To reduce manufacturing costs and to increase the flexibility of systems, the systems getting more and more open. I love the strict separation of control and data plane and that you can influence lots of things better. Running a route engine separate from data plane offers so much possibilities. With SDN this approach is not an exclusive only feature for big core routers. To be competitive you need to implement this way of doing networking even into the mid-range market. Without it you cannot do SDN. ;) And that is what I mean. Networking companies do not offer a totally new technology, they implement Internet proven technology into many switches, routers and security devices. That’s cool and you’re able to do SDN even as small company.
In the end I don’t think SDN is all about a colorful web UI for VLAN provisioning. Creating and managing complex networking environments, needs experts! Doing it yourself with so called “Click-Admins” ends in a catastrophe. Even small companies recognized this, because a stable network is mission critical. Every boss gets very angry then he’s not able to receive and send mails. ;) SDN is built for the experts. The get inspired for new ideas to solve problems. Think about Junipers “Service-Chain” idea or about the new way how’s Google doing networking in his new datacenter. Designed and implemented with experts only. I think all web UIs for SDN are only an add-on for the sales people. ;)
Is Netconf the end all protocol? Probably not. Is IRS? Probably not. The thing we need is something standard across platforms/vendors that can be predictably interacted with. Ultimately I don't see us getting there any time soon. Each SDN product coming out right now pretty much locks you in to one vendor's product. It seems at though OpenFlow has been abandoned for more proprietary crap. So at this point ( and I've said this before) I'm hoping more people go the way Arista have. Offer APIs into their operating system down to the ASIC driver. Allow me the flexibility to develop applications on top of the device.
In my pipe dream each vendor allows you to create your own applications and place them on the device. I can build my own client application with my own capabilities. I can port that client program over to any number of vendors only changing to the code for the API. This will allow software companies or even the vendors themselves to develop unique applications that leverage this openness, or allow engineers with development experience to taylor their network to what they need to accomplish.
Realistically though I think we're a long way off from anything standard and in the meantime we'll just keep watching proprietary system after proprietary system come out claiming to solve some unique problem while providing all the vendor lockin of the big guys's fabrics. So until we reach a point where we (I.E. the engineers) can actually do (Program) some cool stuff with these new innovations I'll remain underwhelmed.
I also think this might be more of a software architecture argument than how you actually get the data from the devices. I just can't agree that Expect is still a good option. It can't be deterministically parsed. So getting useful data from it consistently will always be a struggle.
I do agree the core of the application should be separated from how the device is interacted with. That was part of my original point. If you have the freedom to build your own applications on top of devices you can build the core of your app and the adapt the device interaction to taylor it to different vendors.
As far as the computation of XML vs Expect I think we're splitting hairs here. More than likely we're talking about responses to commands pushed to the device. In which case the returned XML output would be only nominally larger than an Expect output. Also we're talking about maybe milliseconds for a computer to parse that response. And if we're not talking about responses to entered commands then we're talking about getting data back from the device...in which case my original argument stands here. I wan't useable data rather than large text blocks.
In any case, parsing XML is always more deterministic than screen scraping.
There are some realities though -- (1) just because it's XML doesn't mean the vendor won't change the schema, (2) in many years of using Expect, the number of times regular expressions in my parsers needed update from one version of OS to the next for no other reason hasn't really been all that much. Also, whether you are using Expect, XML, Openflow or whatever, you still need to perform regression testing of your programs against the new OS or platform and make adjustments as needed.
Do your parsing with internal subroutines, integrate those functions with a Perl web framework (I used Mojolicious) and the result is JSON/XML-parsed switch data output display, REST api input for provisioning methods to be used across either one or an array of switches. Top it off with DB syncing to track device states and you've got a pretty usable framework.
Granted this module is limited to Cisco devices, but it's a great way to abstract management functions to write applications around, regardless of what your topology really looks like.
This article caught my attention because, while I've considered trying to make a product out of this project, there really doesn't seem to be a market for this thing, at least not one that's super obvious. Most people seem to just write these applications themselves in-house based on their own needs.
I still feel that there is something to be said however for an abstraction layer such as this for your traditional network provisioning requirements on physical switches. VM network provisioning can be and is easily automated by making use of similar interaces provided by the likes of OVS, however that pesky top-of-rack switch that the hypervisor connects to is still in the stone age.