Response: Nothing Works (in Enterprise IT)
Dmitry Perets left a thoughtful comment on my Nothing Works blog post describing why enterprise IT might be even worse than consumer world.
I think another reason for the “Nothing Works” world is that the only true Management Plane separation that exists in our industry is that of the real “human” management. In the medium/large enterprises they (and their interests, KPIs and so on) are very much separated from the technical workforce. And increasingly so, because today the technical workforce might not even be the employees of the same enterprise. They are likely to come from some IT consultancy outsource – degree of separation which makes a true SDN evangelist envious.
So take an example. You – as a private person – buy a piece of consumer goods. Doesn’t really matter what it is and what fruit appears on its back. What matters is that you are the end-user of it. And if using it causes a lot of pain, this will be your pain. So there is at least a tiny chance that you will rethink your choice next time and maybe try looking around or waiting for improvements.
I can immediately see two gotchas:
- Very few people take the time to write a scathing review if they get burned buying a product they only need once, so the rest of us don’t benefit from their horrible experience.
- Sometimes you can’t get a decent product or service because everyone furiously participates in the race to the bottom.
But if you are that manager who decided to purchase a product XYZ for your enterprise, the pain of trying to make it work will not be your pain, it will be the pain of that technical workforce, about whose suffering – frankly – you couldn’t care less. As long as you can report the right PPT bullets (and KPIs) all the way upstream inside your enterprise organization.
That’s we always used very specific requirements when we had to use purchasing managers or public tenders in my previous life1. At the very least you’d get the equipment that has some chance of working (as opposed to whichever layer-3 switch was the cheapest this week). Don’t get me started on how hard it is to write those requirements to ensure you’ll get support from a semi-qualified integration partner.
Of course, if things go really wrong – hours of outage again and again, stuff doesn’t work at all, that kind of things – then the vendor XYZ will eventually be in trouble. Although usually they still can get away with it using one of the standard solutions (“Your technical guys did it all wrong, here is a free technical training for you to avoid this in the future!”, “Yes, but we have this case described in the list of known issues”, “We are sorry, here is a discount that will make it easier for you to reach your cost savings KPIs next quarter” and so on). But most ugly designs can still somehow work with the right amount of static routes and manual interventions, and who cares if the eyes of some “unreasonable employees” bleed while doing this =)
I don’t think I’ve ever seen a vendor get in trouble. If you have interesting counterexamples, please leave a comment or send me an email.
Some people go to bizarre lengths: supposedly someone once specified that the cars they wanted to buy for a government department had to have four circles on the grille 🤷♂️ ↩︎
I have experienced a case where an enterprise network department returned the new networking devices from a well known and respected vendor, and bought gear from a different vendor. Sadly I do not know more details, i.e., how bad it really was, but it must have been devastatingly bad, from the problems I often see accepted in networking gear by the networking departments. (Sorry, I am not in a position to name names regarding this.)
I recall 1999/2000 a major client in financial/insurance industry. They were in the middle of managing Token-Ring/Ethernet/FDDI and some ATM landscape. We spent two years on a major vendor’s Token Ring Switching solution for a 30 floor building. I was in the weeds on that one and still have some detailed diagrams on the “mechanics” of it. Nonetheless, the vendor’s solution was a flop(bugs, interoperable issues, etc)and they took the whole thing back, every switch, and we went full Ethernet switching after that point.
Now that you mention it, ages ago we were "blessed" with an installation where the customer wanted to have ATM LANE (because "ATM is the future") on Catalyst 3000 (or whatever the Kalpana switch was rebranded to) because that was the only Cisco switch that was price-competitive against 3Com.
It was a colossal failure resulting in all switches being RMA-ed for Catalyst 5000s. At least Cisco owned the FUBAR and did everything they could to make the network work for the customer.