One of my readers was listening to the Snabb Switch podcast and started wondering “whether it’s possible to leverage and adopt these bleeding-edge technologies without a substantial staff of savvy programmers?”

Short answer: No. Someone has to do the heavy lifting, regardless of whether you have programmers on-site, outsource the work to contractors, or pay vendors to do it.

Also, someone has to support, troubleshoot, maintain and improve the product over the years to come. This is the also the most-overlooked aspect of any project, particularly homegrown automation solutions. When you roll out your solution into production you’re not done – your job has just started for real.

My reader continued his email by explaining how they work today:

Whenever we adopt bleeding-edge technologies, it’s innovation with training wheels. We try new things but usually with someone holding our hand. There was a recent debate about whether to attempt Openstack internally or via third-party such as Mirantis.

In most cases Innovation with training wheels is the only thing that makes sense, regardless of whether it’s personal growth (read: attend trainings) or introduction of new technologies… unless of course you’re doing something fundamentally new.

Reinventing the wheels (as opposed to having a helping hand that supplies the training wheels) is usually a waste of time - I’ve seen several teams that wasted years (real-time years, not man-years) trying to get OpenStack working from vanilla distro instead of buying a ready-to-use product.

On the other hand, unless you want to work with your consultants (or system integrator or vendor) for years, you must be ready to take over and strip the training wheels eventually.

Products like Cumulus-on-whitebox and Snabb Switches bring opportunities for either highly customized needs or even cost-savings when compared to big-name network vendors. But the argument inevitably comes back to whether or not the cost-savings is worth the risk and headaches of programming and supporting.

It depends on how big you are. You’ll pay the vendor’s R&D and support costs (+ sales and marketing) or you’ll pay your own programmers or contractors. If you’re big enough it makes sense. If you have special needs (high-frequency trading comes to mind) or operate at the very bleeding edge, it might be the only option you have. If you only need a few switches it’s a waste of time.

Also, keep in mind that the infrastructure inevitably becomes a commodity over time. We’re seeing the early stages of that process in networking - data center switches running Linux with a device driver that controls the switching silicon. We had open-source infrastructure including vendor support in reasonably-high-speed x86 packet processing for years (PF_RING, netmap…). The same process has started in bleeding-edge x86-based networking (solutions like 6WIND or Snabb Switch).

Reinventing that infrastructure clearly doesn’t make sense - it’s like writing your own CRM instead of using SalesForce (or another similar product).

My reader concluded his email by saying:

However, I refuse to fall into that narrow-minded and lazy mindset of just going with the usual big-name, incumbent vendor over and over (even after the CIO asks if there are opportunities to reduce costs).

If you need a few instances of a product, go with an incumbent vendor. It’s not worth risking your whole data center just to get a slightly cheaper pair of load balancers or firewalls. If you need a few thousand boxes, it makes sense to look around. In between there’s the usual gray area.

Also keep in mind that reducing CapEx (= buying a product) often increases OpEx (= building a product), and you have to be very careful and realistic in your estimates. The costs of building a product are usually underestimated by order of magnitude, particularly by the people who never built one (see also Dunning-Kruger effect), and the costs of maintaining a product (particularly syncing your fork when the underlying codebase evolves) are usually conveniently ignored.

Finally, doing the same thing over and over and expecting reduced costs is close the well-known definition of insanity. The real cost-savings come from changing your application development and deployment processes and removing the unnecessary kludges and reverse engineering that happen between Dev and Prod.