Another interesting question I got from an ipSpace.net subscriber:
Assuming we can simplify the physical network when using overlay virtual network solutions like VMware NSX, do we really need datacenter switches (example: Cisco Nexus instead of Catalyst product line) to implement the underlay?
Let’s recap what we really need to run VMware NSX:
- Multiple VRFs;
- VLANs stretched across at least two ToR switches (or IP host routing hacks), potentially combined with MLAG.
- Jumbo frames to support NSX-generated VXLAN packets carrying full-sized Ethernet frames.
Next question: how big is your fabric?
If you’re small enough for two switches, go with a solution that uses two independent switches - you really don’t want to run your whole infrastructure on a glorified stackable switch (regardless of whether it’s called VSS, IRF or VCF). Also, keep it as simple as possible - kick out any vendor that tries to sell you VXLAN with EVPN and BGP.
If you need more than two switches, you’ll have to build a fabric (I would opt for a leaf-and-spine fabric) that supports multiple VRFs and VLANs spanning more than one ToR switch. As most vendors abandoned the proprietary fabric craze, you have two options:
- Traditional bridging with leaf-to-spine MLAG;
- VXLAN-based overlay networking.
To recap - you’re looking for switches that:
- Do MLAG with independent control planes;
- Support multiple VRFs;
- Support VXLAN-based layer-2 transport;
- (Optionally) Have EVPN control plane.
Now that you have a list of requirements, it should be easy to figure out what switches meet them. For more details, watch Data Center Fabric Architectures, Leaf-and-Spine Architectures, EVPN Deep Dive, and Designing Private Cloud Infrastructure webinars.
2020-03-11: Added jumbo frames requirement based on feedback from Jerome Catrouillet. Thank you!