VXLAN Hardware Gateway Overview

One of my readers stumbled upon blog post from 2011 explaining the potential implementations of VXLAN hardware gateways, and asked me if that information is still relevant.

I knew that I’d included tons of information in the Data Center Fabrics and VXLAN Deep Dive webinars, but couldn’t find anything on the web, so I decided to fix that in 2015.

2020 Update

As expected, the information published in 2015 did not age well. Here’s a short summary of the current state of data center switches; you might still find some useful information in the rest of the blog post.

  • Data center switches from all major vendors support VXLAN.
  • Arista, Cisco, Cumulus, and Juniper support EVPN control plane, multicast-based VXLAN, and statically configured ingress replication. I did not check HP or Dell, and Brocade is gone anyway.
  • Multicast-based VXLAN became a niche solution. While you can still configure static ingress replication lists, everyone is pushing EVPN control plane.
  • VXLAN routing is available on switches using vendor silicon (most Cisco Nexus switches, Juniper QFX10K), and on switches using the following merchant silicon: Trident-2+, Trident-3 or later, Jericho (deep buffer), Mellanox Spectrum. Broadcom Tomahawk does not support VXLAN routing; switch vendors might use tricks like recirculation or multi-stage switching to get it done (more info in the updated VXLAN webinar).
  • OVSDB is obsolete, as is VMware NSX-V. VMware NSX-T integration with hardware gateways uses EVPN.

Have I missed something? Please leave a comment!

Original Blog Post

Here’s a brief overview of what individual vendors’ hardware gateways (ToR switches) can or cannot do (to the best of my knowledge).

Vendor Multicast VXLAN  OVSDB VXLAN Routing EVPN
Arista 7150 only
Brocade
Cisco Nexus 7K/9K
Citrix Netscaler
Cumulus
Dell
F5 BIG-IP
HP ✅ (*) ✅ (*)
Juniper ✅ (*) MX and EX9200

Notes

  • Nuage (or Alcatel Lucent) has Virtualized Services Gateway, which is another VXLAN gateway, but as I couldn’t find any documentation on Nuage or Alcatel Lucent web site (and the VSP documentation is behind a regwall), it’s not on the list. Five years later, some vendors still haven’t got the memo.
  • A10 Networks is another vendor who hasn't got that same memo yet.
  • HP has VXLAN support on several Data Center switches, but according to the configuration guide(s) at the moment only 5930 supports multicast VXLAN and OVSDB. Please check HP documentation for up-to-date status;
  • Juniper QFX5100, QFX10K, EX9200 and MX routers support VXLAN and OVSDB. QFX10K does not support multicast VXLAN (yet). Only EX9200 and MX routers support VXLAN routing.
  • Multicast VXLAN support allows ToR switch to interact with Cisco Nexus 1000V and pre-NSX VMware VXLAN implementations;
  • OVSDB is the protocol used by VMware NSX for Multiple Hypervisors to configure ToR switches. We don’t know yet what VMware will use when support for hardware gateways will be added to NSX for vSphere, but I wouldn’t be surprised if they were to use OVSDB;
  • VXLAN routing is tricky – more details here and here;
  • EVPN control plane enables large L2 fabrics built on top of VXLAN and controller federation;

For more details, go watch the two webinars (links above).

20 comments:

  1. nice overview! What about the idea using an ADC such as A10 Networks Thunder ADC as a VXLAN Gateway?
    Replies
    1. No public documentation ==> No mention. Nuff said ;)
  2. NSX 6.2 added OVSDB-based integration with Hardware VTEPs. It is still listed as a “technology preview” however, so not supported officially, yet. I believe Arista was doing demos of it at VMworld
    Replies
    1. to clarify — added OVSDB-based integration for NSX-V
  3. All of the F5 modules (not just LTM) support VXLAN since it is actually part of their "TMOS" (OS).
  4. was multicast vxlan considered a benefit or a legacy/inheritance from the past? if it's a benefit, what about the unicast mode?
    Replies
    1. Unicast VXLAN needs a control plane - OVSDB or EVPN.
    2. It seems Citrix has another way to do Unicast VXLAN (which does not scale):
      Unicast mode: In this mode, you specify the IP addresses of VTEPs while configuring a VXLAN on a NetScaler ADC. The NetScaler ADC sends broadcast, multicast, and unknown unicast frames over Layer 3 to all VTEPs of this VXLAN.
      http://docs.citrix.com/en-us/netscaler/11/networking/vxlans.html
  5. f5 added OVSDB support in their latest release (v12) of TMOS, and it - as well as VxLAN routing, gateway, and multicast functions - is available on their whole BIG-IP portfolio.
    Replies
    1. public docs:

      https://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/tmos-implementations-12-0-0/9.html

    2. Thank you. Much appreciated! Updated the blog post.
  6. HP's 5930 latest release include OVSDB in the latest firmware release.
    It was announced on vmware word 2015. The product brochure needs to be updated as it makes no mention of it.

    http://www8.hp.com/us/en/hp-news/press-release.html?id=2068320

    Disclaimer: I work for HP
    Replies
    1. Thanks for the update. Is it GA/shipping or was it just announced?
    2. Already released and available. I think that even the previous version already supported OVSDB.

      5930_7.10.R2418P01

      Release notes:
      http://h20564.www2.hpe.com/portal/site/hpsc/public/kb/docDisplay/?docId=c04719279

      Firmware
      https://h10145.www1.hp.com/downloads/SoftwareReleases.aspx?ProductNumber=JG726A&lang=&cc=&prodSeriesId=
    3. Thank you. Much appreciated! Updated the blog post.
  7. Ivan: you kind of just glancing over EVPN as if it's not such a big deal, but you're probably deeply satisfied that your beloved MPLS VPN concepts is rearing it's beautiful head here again ;)
  8. Juniper does have evpn support on MX&EX9200 right? Or am I missing something here http://pathfinder.juniper.net/feature-explorer/search.html#q=evpn
    Replies
    1. They do have EVPN, but not with VXLAN encapsulation (just checked Junos 14.2 and 15.1 manuals, but maybe I'm missing something).
    2. I think this is available starting in 14.2R4 which was released recently. I haven't tested this out. http://www.juniper.net/techpubs/en_US/junos14.2/information-products/topic-collections/release-notes/14.2/junos-release-notes-14.2.pdf

      MX Series routers can use EVPN with VXLAN encapsulation to provide
      Layer 2 connectivity for end stations within a Virtualized Network (VN) .
    3. Yep, the 14.2R4 release notes talk about that, but there's no mention of VXLAN in EVPN documentation for 14.2 (or Junos Latest), or mention of EVPN in VXLAN/OVSDB documentation.

      Looks like we'll have to wait a bit to get this feature properly documented.
  9. Hi Ivan,

    A couple of updates...

    1. Cisco Nexus 7000 supports EVPN (http://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/nx-os/vxlan/configuration/guide/b_NX-OS_VXLAN_Configuration_Guide/configuring_vxlan_bgp_evpn.html)

    2. HP supports multicast flooding for VXLAN, it's available in the 5930 VXLAN Configuration guide (haven't checked the 7900/12900).

    3. HP also supports a VXLAN control plane, but it is proprietary based on IS-IS rather than EVPN (available on 12900). EVPN support is coming CY2016 from what I have been told.
    Replies
    1. Thank you. Much appreciated! Updated the blog post. BTW, the IS-IS thingy is also available on 5930 (just checked), and multicast/OVSDB is not available on 7900 (at least not as of last Sunday ;).
  10. Alcatel-Lucent and Alcatel-Lucent Enterprise are two different companies now and are still going through mitosis. ALU (Nokia) got the service provider stuff and ALE got the campus LAN/Wifi and DC stuff (OmniX). I'd expect a rebrand soon since they split nearly 12 months ago.

    ALE's OmniSwitch manuals are online. Make sure you hit enterprise.alcatel-lucent.com before trying to get to support. I believe only the OS6900-Q32,X72 support VXLAN termination. The -X20,X40,T20,T40 don't and never will (earlier generation Trident II). Quick skim -- seems to support unicast and multicast VXLAN.

    About 6 months ago Avaya's VSP platform documentation was available, but their site was/is frustratingly hard to navigate. I just gave it a 10 minute try and gave up. IIRC, the VSP 7200 & 8400 support VXLAN, but it's very early implementation.

    Both Avaya and ALE's switches are Broadcom based, which means they both suffer from the problem of not having the right forwarding pipeline stage ordering/count to put a router interface inside a VXLAN VTEP or inside an SPB I-SID. Avaya solves it with some loopback traces on the board (good luck figuring out specs). ALE solves it by saying "use a loopback cable in a link-agg configuration".

    The VXLAN Routing "yes" for HP gives me pause. I believe a lot of HP's 1U/2U lineup is Trident II based. While ComWare may support the construct, the hardware might not.
    Replies
    1. The information about ALE OmniSwitch is accurate. The OS6900-Q32 and OS6900-X72 both provide VXLAN L2 Gateway in hardware (BCM Trident II). Public documentation is at: http://enterprise.alcatel-lucent.com/assets/documents/OmniSwitch_7_DataCenter_Switching_Guide_Rev-F_EN.pdf or http://enterprise.alcatel-lucent.com/includes/documentlinkPostEloq.cfm?id=22516 . You might want to fast forward to chapter 7, which details what is supported with regards to VXLAN. One should notice that ALE is the only vendor that can provide visibility into the VXLAN overlays with all OS6900/OS10K (VM / VXLAN Snooping) detailed in chapter 8. ALE as also published a whitepaper on the interoperability with VMware NSX (v6.1 was tested) here: http://enterprise.alcatel-lucent.com/includes/documentlinkPostEloq.cfm?id=26782 If you have any questions on that, feel free to reach out to me on Twitter @bennye_hh
  11. HP FF 5930 supports VXLAN and OVSDB both.
    A multicast enabled underlay network is not required for HP VXLAN solutions .
    Please refer:
    http://www8.hp.com/h20195/v2/getpdf.aspx/4AA5-7065ENW.pdf?ver=1.0
  12. The ALU/Nuage box is called the 7850 VSG. The data sheet can be found here:
    http://www.nuagenetworks.net/wp-content/uploads/2014/11/NP2013102921EN_V2_NN_7850_VSG_Datasheet1.pdf

    From my memory it supports VXLAN w/EVPN control plane since that's what Nuage uses, not sure about OVSDB. It's Broadcom based so probably doesn't do VXLAN routing.

    EVPN w/VXLAN encap is coming on the QFX switches like the 5100 but isn't available yet.
    Replies
    1. Hi!

      I know what ALU/Nuage box does, but as I cannot get to publicly available documentation, it's not in the table.

      Some vendors might not like this stance. They are always free to change their documentation policy ;)

      Best, Ivan
    2. As shown in Ivan's webinar http://www.ipspace.net/Scaling_Overlay_Virtual_Networks the Nuage 7850 VSG does support VXLAN routing, even using the Broadcom Trident II. It does not use OVSDB
  13. http://docs.citrix.com/en-us/netscaler/11/networking/vxlans.html
    Replies
    1. Added. Thank you! It's really nice to see more appliance vendors having native VXLAN support!
  14. HP 5930 does not support VXLAN Routing. You need a loopback cable to get the exVXLANed packages back into the switch.
    HP 5930 does not support EVPN
    HP 5930 does not support HP´s EVI, i.e. GRE tunnels and IS-IS
  15. Arista does support Multicast VXLAN but not on all platforms specifically not for 7050X series.
  16. To update my own comment. with latest EOS release EOS-4.15.2F Arista now supports Multicast VXLAN for platforms like 7050X, 7250X and 7300X. Haven't tested it though yet.
  17. Juniper just release the 14.1X53-D30 for QFX5100 that include EVPN VxLAN and now they have documentation for it
    http://www.juniper.net/techpubs/en_US/junos14.1/topics/topic-map/sdn-vxlan.html
    (if you can call this mess a documentation ) so you can add QFX5100 and MX to the EVPN section
    Replies
    1. The document you cited contains no mention of EVPN. Still not documented.
    2. Better doc: "Juniper Networks EVPN Implementation for Next-Generation Data Center Architectures"
      http://www.juniper.net/assets/us/en/local/pdf/whitepapers/2000606-en.pdf
  18. Besides hardware switch. Under Linux, OVS can bridge to VXLAN. A Windows base open source tool UBridge which can be run on most kinds of Windows platform (XP/Vista/Win 7/8/ Win servers etc.) also allows the direct bridge of standalone Windows machine to the VXLAN. Pls refer to following link for detail:

    http://www.techezone.com/question/direct-bridging-of-windows-platform-to-the-vxlan/
  19. what's the status as of today instead? (May 2017) ?
    Replies
    1. Different ;) Several vendors support VXLAN routing and EVPN.
  20. Are VXLAN hardware gateway and OVSDB still relevant? Could it be that it is no longer supported in Vmware NSX-T?

    Replies
    1. Thanks for the poke. Updated the blog post.

Add comment
Sidebar