I’ve spent the last few days with a fantastic group of highly skilled networking engineers (can’t share the details, but you know who you are) discussing the topics I like most: BGP, MPLS, MPLS Traffic Engineering and IPv6 in Service Provider environment.

One of the problems we were trying to solve was a clean split of a POP into two sites, retaining redundancy without adding too much extra equipment. The strive for maximum redundancy nudged me to propose the unimaginable: layer-2 interconnect between four tightly controlled routers running BGP, but even that got shot down with a memorable quote from the senior network architect:

I would always prefer slightly reduced redundancy over increased failure domain.

In the end, we figured out that without the layer-2 interconnect our design can’t survive a particular rare combination of two simultaneous failures, but even there we found a workaround that allowed us to retain slightly reduced functionality after a total failure of those two critical components without resorting to stretched subnets. As they say, where there’s will, there’s way.

During the evenings we kept discussing various “interesting” scenarios we’ve seen in the past, including people who tried to cram way too many features into a single router (like trying to run IPSec and L2TP inside a VRF in an ASR or VTI-over-DMVPN on a 2900). I will definitely use the very short explanation I got in some future discussions:

People need to understand that even though they have a Swiss army knife, they can’t use all of its features at the same time.

Giant Swiss Army Knife from ThinkGeek

And the last one probably needs no introductory context:

You need high pressure to get diamonds. Without it, you’re left with coal.