I had a nice chat with Doug Gourlay from Arista during the Interop Las Vegas and he made an interesting remark along the lines of “in leaf-and-spine fabrics it doesn’t make sense to use redundant supervisors in switches – they cause more problems than they solve.”
As always, in the end it all depends on your environment and use case, but he definitely has a point; good engineering always works better than a heap of kludges.
Going back 20 years
In early 90s we didn’t have redundant supervisors (or CPU boards or route processors or whatever they were called). Each box had a single CPU and if you wanted to build a resilient network you did a proper network design – for each function you needed in the network (example: core or aggregation layer) you used two (or more) boxes connected with reasonably fast-converging routing protocol. Problem solved.
Carrier Grade Marketing
Are you old enough to remember the original Internet (and dot.com) bubble? In those days every telecom thought they could find gold nuggets lying in plain sight on the magical plains of planet Internet. Unfortunately, they forgot to leave their old mentality at home when joining the gold rush.
Voice switches (the gear telecom engineers were used to deal with in those days) had all sorts of redundancy. After all, you cannot connect a dumb phone to two voice switches, so you better have a switch that can never crash. Internet is slightly different – good IP-based architectures always relied on smart edge and simple core (virtualization vendors needed more than 10 years to figure that out, but that’s beyond the point).
Regardless of proven facts and working best practices, telecoms wanted to have box-level feature parity between what they knew and what they planned to buy, and networking vendors delivered what the customer wanted – more and more complex boxes with built-in hardware redundancy and all sorts of failover mechanisms, including SSO, NSF and ISSU.
The links in the previous paragraph point to Cisco’s web site, but I’m not trying to pick on Cisco. Every vendor of high-end gear, including Cisco, Juniper, HP and Arista has similar feature set.
With all the great redundancy features being implemented to improve vendors’ chances in carrier market, it was time to reap the benefits of that investment. Next stop: enterprise networks.
Is it all just hype?
It depends. You can always implement redundant or resilient solutions. Resilient is usually better than redundant, but there are cases where boxes with redundant internal architecture come handy due to the tradeoffs you might have to make to implement a resilient design.
Example: campus networks. In campus networks you cannot afford to lose a whole building, but it might be OK to lose half a floor.
A resilient design would use two core switches and an access switch (or more) per floor. Ideally they’d run a fast-converging routing protocol.
In reality, you’re often asked to implement bridging across a whole building, and as there are no standard layer-2 fabric solutions, you have to use spanning tree (and lose half the uplink bandwidth in the process) or MLAG (which increases the complexity of your design). Also, managing tons of small switches manually (because the network management software almost never does everything its vendor has promised) becomes a royal pain.
A core switch with redundant architecture definitely seems like a better option, but do keep in mind that you’ve just traded visible complexity that you understand and are thus able to troubleshoot, with hidden complexity.
Data center environments
Data center networks are always considered to be mission critical, and it makes perfect sense to buy an insurance policy in form of redundant hardware architecture, right? Well, no.
Unless you’re Google or Facebook and can afford to lose 50 servers on a ToR switch reload you probably have dual-homed servers connected to two ToR switches, right? Losing one of those switches hurts (you lose half the bandwidth), but not too much. No wonder no ToR switches have redundant supervisors (Juniper’s QFX 5100 is an interesting semi-exception: they can run two copies of Junos as virtual machines on the same CPU).
Losing one of two core switches is a major disaster – half the core switching bandwidth is lost. How do you cope with that? You buy switches with two supervisor boards and hope the internal hardware and software redundancy works as expected.
I’ve seen data center network designs with a single core switch (“we don’t need two because we bought a fully redundant box”). Don’t ever do that, you’ll end up with a redundantly engineered single point of failure.
Now imagine you replace two humongous core switches with a spine layer having 4 or 8 fixed or modular switches. All of a sudden losing a spine switch doesn’t hurt that much. Welcome to the wonderful world of proper network design ;)
Building a data center fabric?
If you’re new to leaf-and-spine designs, the Clos Fabrics webinar will get you started, and if you’re interested in more than one webinar, the Data Center Webinars package or yearly subscription might be the best option.
Finally, in always available for an online consulting session.