Another interesting question I got from an ipSpace.net subscriber:

Assuming we can simplify the physical network when using overlay virtual network solutions like VMware NSX, do we really need datacenter switches (example: Cisco Nexus instead of Catalyst product line) to implement the underlay?

Let’s recap what we really need to run VMware NSX:

• Multiple VRFs;
• VLANs stretched across at least two ToR switches (or IP host routing hacks), potentially combined with MLAG.
• Jumbo frames to support NSX-generated VXLAN packets carrying full-sized Ethernet frames.

Next question: how big is your fabric?

If you’re small enough for two switches, go with a solution that uses two independent switches - you really don’t want to run your whole infrastructure on a glorified stackable switch (regardless of whether it’s called VSS, IRF or VCF). Also, keep it as simple as possible - kick out any vendor that tries to sell you VXLAN with EVPN and BGP.

If you need more than two switches, you’ll have to build a fabric (I would opt for a leaf-and-spine fabric) that supports multiple VRFs and VLANs spanning more than one ToR switch. As most vendors abandoned the proprietary fabric craze, you have two options:

• Traditional bridging with leaf-to-spine MLAG;
• VXLAN-based overlay networking.

Last time I looked at my calendar it was 2020. Maybe we should stop using the MLAG+STP hacks even when you find them in old vendor marketing materials.

To recap - you’re looking for switches that:

• Do MLAG with independent control planes;
• Support multiple VRFs;
• Support VXLAN-based layer-2 transport;
• (Optionally) Have EVPN control plane.

Now that you have a list of requirements, it should be easy to figure out what switches meet them. For more details, watch Data Center Fabric Architectures, Leaf-and-Spine Architectures, EVPN Deep Dive, and Designing Private Cloud Infrastructure webinars.

2020-03-11: Added jumbo frames requirement based on feedback from Jerome Catrouillet. Thank you!