… updated on Thursday, March 31, 2022 16:03 UTC
VXLAN Hardware Gateway Overview
One of my readers stumbled upon blog post from 2011 explaining the potential implementations of VXLAN hardware gateways, and asked me if that information is still relevant.
I knew that I’d included tons of information in the Data Center Fabrics and VXLAN Deep Dive webinars, but couldn’t find anything on the web, so I decided to fix that in 2015.
2020 Update
As expected, the information published in 2015 did not age well. Here’s a short summary of the current state of data center switches; you might still find some useful information in the rest of the blog post.
- Data center switches from all major vendors support VXLAN.
- Arista, Cisco, Cumulus, and Juniper support EVPN control plane, multicast-based VXLAN, and statically configured ingress replication. I did not check HP or Dell, and Brocade is gone anyway.
- Multicast-based VXLAN became a niche solution. While you can still configure static ingress replication lists, everyone is pushing EVPN control plane.
- VXLAN routing is available on switches using vendor silicon (most Cisco Nexus switches, Juniper QFX10K), and on switches using the following merchant silicon: Trident-2+, Trident-3 or later, Jericho (deep buffer), Mellanox Spectrum. Broadcom Tomahawk does not support VXLAN routing; switch vendors might use tricks like recirculation or multi-stage switching to get it done (more info in the updated VXLAN webinar).
- OVSDB is obsolete, as is VMware NSX-V. VMware NSX-T integration with hardware gateways uses EVPN.
Have I missed something? Please leave a comment!
Original Blog Post
Here’s a brief overview of what individual vendors’ hardware gateways (ToR switches) can or cannot do (to the best of my knowledge).
Vendor | Multicast VXLAN | OVSDB | VXLAN Routing | EVPN |
---|---|---|---|---|
Arista | ✅ | ✅ | 7150 only | |
Brocade | ✅ | |||
Cisco | ✅ | ✅ | ✅ | Nexus 7K/9K |
Citrix Netscaler | ✅ | |||
Cumulus | ✅ | |||
Dell | ✅ | |||
F5 BIG-IP | ✅ | ✅ | ✅ | |
HP | ✅ (*) | ✅ (*) | ✅ | |
Juniper | ✅ (*) | ✅ | MX and EX9200 |
Notes
- Nuage (or Alcatel Lucent) has Virtualized Services Gateway, which is another VXLAN gateway, but as I couldn’t find any documentation on Nuage or Alcatel Lucent web site (and the VSP documentation is behind a regwall), it’s not on the list. Five years later, some vendors still haven’t got the memo.
- A10 Networks is another vendor who hasn't got that same memo yet.
- HP has VXLAN support on several Data Center switches, but according to the configuration guide(s) at the moment only 5930 supports multicast VXLAN and OVSDB. Please check HP documentation for up-to-date status;
- Juniper QFX5100, QFX10K, EX9200 and MX routers support VXLAN and OVSDB. QFX10K does not support multicast VXLAN (yet). Only EX9200 and MX routers support VXLAN routing.
- Multicast VXLAN support allows ToR switch to interact with Cisco Nexus 1000V and pre-NSX VMware VXLAN implementations;
- OVSDB is the protocol used by VMware NSX for Multiple Hypervisors to configure ToR switches. We don’t know yet what VMware will use when support for hardware gateways will be added to NSX for vSphere, but I wouldn’t be surprised if they were to use OVSDB;
- VXLAN routing is tricky – more details here and here;
- EVPN control plane enables large L2 fabrics built on top of VXLAN and controller federation;
For more details, go watch the two webinars (links above).
Unicast mode: In this mode, you specify the IP addresses of VTEPs while configuring a VXLAN on a NetScaler ADC. The NetScaler ADC sends broadcast, multicast, and unknown unicast frames over Layer 3 to all VTEPs of this VXLAN.
http://docs.citrix.com/en-us/netscaler/11/networking/vxlans.html
https://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/tmos-implementations-12-0-0/9.html
It was announced on vmware word 2015. The product brochure needs to be updated as it makes no mention of it.
http://www8.hp.com/us/en/hp-news/press-release.html?id=2068320
Disclaimer: I work for HP
5930_7.10.R2418P01
Release notes:
http://h20564.www2.hpe.com/portal/site/hpsc/public/kb/docDisplay/?docId=c04719279
Firmware
https://h10145.www1.hp.com/downloads/SoftwareReleases.aspx?ProductNumber=JG726A&lang=&cc=&prodSeriesId=
MX Series routers can use EVPN with VXLAN encapsulation to provide
Layer 2 connectivity for end stations within a Virtualized Network (VN) .
Looks like we'll have to wait a bit to get this feature properly documented.
A couple of updates...
1. Cisco Nexus 7000 supports EVPN (http://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/nx-os/vxlan/configuration/guide/b_NX-OS_VXLAN_Configuration_Guide/configuring_vxlan_bgp_evpn.html)
2. HP supports multicast flooding for VXLAN, it's available in the 5930 VXLAN Configuration guide (haven't checked the 7900/12900).
3. HP also supports a VXLAN control plane, but it is proprietary based on IS-IS rather than EVPN (available on 12900). EVPN support is coming CY2016 from what I have been told.
ALE's OmniSwitch manuals are online. Make sure you hit enterprise.alcatel-lucent.com before trying to get to support. I believe only the OS6900-Q32,X72 support VXLAN termination. The -X20,X40,T20,T40 don't and never will (earlier generation Trident II). Quick skim -- seems to support unicast and multicast VXLAN.
About 6 months ago Avaya's VSP platform documentation was available, but their site was/is frustratingly hard to navigate. I just gave it a 10 minute try and gave up. IIRC, the VSP 7200 & 8400 support VXLAN, but it's very early implementation.
Both Avaya and ALE's switches are Broadcom based, which means they both suffer from the problem of not having the right forwarding pipeline stage ordering/count to put a router interface inside a VXLAN VTEP or inside an SPB I-SID. Avaya solves it with some loopback traces on the board (good luck figuring out specs). ALE solves it by saying "use a loopback cable in a link-agg configuration".
The VXLAN Routing "yes" for HP gives me pause. I believe a lot of HP's 1U/2U lineup is Trident II based. While ComWare may support the construct, the hardware might not.
A multicast enabled underlay network is not required for HP VXLAN solutions .
Please refer:
http://www8.hp.com/h20195/v2/getpdf.aspx/4AA5-7065ENW.pdf?ver=1.0
http://www.nuagenetworks.net/wp-content/uploads/2014/11/NP2013102921EN_V2_NN_7850_VSG_Datasheet1.pdf
From my memory it supports VXLAN w/EVPN control plane since that's what Nuage uses, not sure about OVSDB. It's Broadcom based so probably doesn't do VXLAN routing.
EVPN w/VXLAN encap is coming on the QFX switches like the 5100 but isn't available yet.
I know what ALU/Nuage box does, but as I cannot get to publicly available documentation, it's not in the table.
Some vendors might not like this stance. They are always free to change their documentation policy ;)
Best, Ivan
HP 5930 does not support EVPN
HP 5930 does not support HP´s EVI, i.e. GRE tunnels and IS-IS
http://www.juniper.net/techpubs/en_US/junos14.1/topics/topic-map/sdn-vxlan.html
(if you can call this mess a documentation ) so you can add QFX5100 and MX to the EVPN section
http://www.juniper.net/assets/us/en/local/pdf/whitepapers/2000606-en.pdf
http://www.techezone.com/question/direct-bridging-of-windows-platform-to-the-vxlan/
Are VXLAN hardware gateway and OVSDB still relevant? Could it be that it is no longer supported in Vmware NSX-T?
Thanks for the poke. Updated the blog post.