Build the Next-Generation Data Center
6 week online course starting in spring 2017

VXLAN Hardware Gateway Overview

One of my readers stumbled upon a 4-year-old blog post explaining the potential implementations of VXLAN hardware gateways, and asked me if that information is still relevant.

I knew that I’d included tons of information in the Data Center Fabrics and VXLAN Deep Dive webinars, but couldn’t find anything on the web, so let’s fix that.

Update 2015-09-14: Added more information on Juniper VXLAN support. Added F5 LTM to the table.

Update 2015-09-15: F5 has OVSDB support and does VXLAN routing. Nexus 7K has EVPN. HP 5930 has multicast VXLAN and OVSDB.

Update 2015-09-16: Cumulus Linux has OVSDB support.

Update 2015-09-18: Citrix Netscaler supports multicast VXLAN or statically-configured unicast VXLAN.

Here’s a brief overview of what individual vendors’ hardware gateways (ToR switches) can or cannot do (to the best of my knowledge – please write a comment if I missed anything).

If you stumble upon this post 4 years from now, it's probably also outdated, but the webinars won't be.

Vendor

Multicast
VXLAN

OVSDB
Support

VXLAN
Routing

EVPN Control
Plane

Arista

Yes

Yes

7150 only

Brocade

Yes

Cisco

Yes

Yes

Nexus 7K/9K

Citrix Netscaler

Yes

Cumulus

Yes

Dell

Yes

F5 BIG-IP

Yes

Yes

Yes

HP

Yes (*)

Yes (*)

Yes

Juniper

Yes
(not QFX10K)

Yes

MX & EX9200

Notes

  • Nuage (or Alcatel Lucent) has Virtualized Services Gateway, which is another VXLAN gateway, but as I couldn’t find any documentation on Nuage or Alcatel Lucent web site (and the VSP documentation is behind a regwall), it’s not on the list. Five years later, some vendors still haven’t got the memo.
  • A10 Networks is another vendor who hasn't got that same memo yet.
  • HP has VXLAN support on several Data Center switches, but according to the configuration guide(s) at the moment only 5930 supports multicast VXLAN and OVSDB. Please check HP documentation for up-to-date status;
  • Juniper QFX5100, QFX10K, EX9200 and MX routers support VXLAN and OVSDB. QFX10K does not support multicast VXLAN (yet). Only EX9200 and MX routers support VXLAN routing.
  • Multicast VXLAN support allows ToR switch to interact with Cisco Nexus 1000V and pre-NSX VMware VXLAN implementations;
  • OVSDB is the protocol used by VMware NSX for Multiple Hypervisors to configure ToR switches. We don’t know yet what VMware will use when support for hardware gateways will be added to NSX for vSphere, but I wouldn’t be surprised if they were to use OVSDB;
  • VXLAN routing is tricky – more details here and here;
  • EVPN control plane enables large L2 fabrics built on top of VXLAN and controller federation;

For more details, go watch the two webinars (links above).

38 comments:

  1. nice overview! What about the idea using an ADC such as A10 Networks Thunder ADC as a VXLAN Gateway?

    ReplyDelete
    Replies
    1. No public documentation ==> No mention. Nuff said ;)

      Delete
  2. NSX 6.2 added OVSDB-based integration with Hardware VTEPs. It is still listed as a “technology preview” however, so not supported officially, yet. I believe Arista was doing demos of it at VMworld

    ReplyDelete
    Replies
    1. to clarify — added OVSDB-based integration for NSX-V

      Delete
  3. All of the F5 modules (not just LTM) support VXLAN since it is actually part of their "TMOS" (OS).

    ReplyDelete
  4. was multicast vxlan considered a benefit or a legacy/inheritance from the past? if it's a benefit, what about the unicast mode?

    ReplyDelete
    Replies
    1. Unicast VXLAN needs a control plane - OVSDB or EVPN.

      Delete
    2. It seems Citrix has another way to do Unicast VXLAN (which does not scale):
      Unicast mode: In this mode, you specify the IP addresses of VTEPs while configuring a VXLAN on a NetScaler ADC. The NetScaler ADC sends broadcast, multicast, and unknown unicast frames over Layer 3 to all VTEPs of this VXLAN.
      http://docs.citrix.com/en-us/netscaler/11/networking/vxlans.html

      Delete
  5. f5 added OVSDB support in their latest release (v12) of TMOS, and it - as well as VxLAN routing, gateway, and multicast functions - is available on their whole BIG-IP portfolio.

    ReplyDelete
    Replies
    1. public docs:

      https://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/tmos-implementations-12-0-0/9.html

      Delete
    2. Thank you. Much appreciated! Updated the blog post.

      Delete
  6. HP's 5930 latest release include OVSDB in the latest firmware release.
    It was announced on vmware word 2015. The product brochure needs to be updated as it makes no mention of it.

    http://www8.hp.com/us/en/hp-news/press-release.html?id=2068320

    Disclaimer: I work for HP

    ReplyDelete
    Replies
    1. Thanks for the update. Is it GA/shipping or was it just announced?

      Delete
    2. Already released and available. I think that even the previous version already supported OVSDB.

      5930_7.10.R2418P01

      Release notes:
      http://h20564.www2.hpe.com/portal/site/hpsc/public/kb/docDisplay/?docId=c04719279

      Firmware
      https://h10145.www1.hp.com/downloads/SoftwareReleases.aspx?ProductNumber=JG726A&lang=&cc=&prodSeriesId=

      Delete
    3. Thank you. Much appreciated! Updated the blog post.

      Delete
  7. Ivan: you kind of just glancing over EVPN as if it's not such a big deal, but you're probably deeply satisfied that your beloved MPLS VPN concepts is rearing it's beautiful head here again ;)

    ReplyDelete
  8. Juniper does have evpn support on MX&EX9200 right? Or am I missing something here http://pathfinder.juniper.net/feature-explorer/search.html#q=evpn

    ReplyDelete
    Replies
    1. They do have EVPN, but not with VXLAN encapsulation (just checked Junos 14.2 and 15.1 manuals, but maybe I'm missing something).

      Delete
    2. I think this is available starting in 14.2R4 which was released recently. I haven't tested this out. http://www.juniper.net/techpubs/en_US/junos14.2/information-products/topic-collections/release-notes/14.2/junos-release-notes-14.2.pdf

      MX Series routers can use EVPN with VXLAN encapsulation to provide
      Layer 2 connectivity for end stations within a Virtualized Network (VN) .

      Delete
    3. Yep, the 14.2R4 release notes talk about that, but there's no mention of VXLAN in EVPN documentation for 14.2 (or Junos Latest), or mention of EVPN in VXLAN/OVSDB documentation.

      Looks like we'll have to wait a bit to get this feature properly documented.

      Delete
  9. Hi Ivan,

    A couple of updates...

    1. Cisco Nexus 7000 supports EVPN (http://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/nx-os/vxlan/configuration/guide/b_NX-OS_VXLAN_Configuration_Guide/configuring_vxlan_bgp_evpn.html)

    2. HP supports multicast flooding for VXLAN, it's available in the 5930 VXLAN Configuration guide (haven't checked the 7900/12900).

    3. HP also supports a VXLAN control plane, but it is proprietary based on IS-IS rather than EVPN (available on 12900). EVPN support is coming CY2016 from what I have been told.

    ReplyDelete
    Replies
    1. Thank you. Much appreciated! Updated the blog post. BTW, the IS-IS thingy is also available on 5930 (just checked), and multicast/OVSDB is not available on 7900 (at least not as of last Sunday ;).

      Delete
  10. Alcatel-Lucent and Alcatel-Lucent Enterprise are two different companies now and are still going through mitosis. ALU (Nokia) got the service provider stuff and ALE got the campus LAN/Wifi and DC stuff (OmniX). I'd expect a rebrand soon since they split nearly 12 months ago.

    ALE's OmniSwitch manuals are online. Make sure you hit enterprise.alcatel-lucent.com before trying to get to support. I believe only the OS6900-Q32,X72 support VXLAN termination. The -X20,X40,T20,T40 don't and never will (earlier generation Trident II). Quick skim -- seems to support unicast and multicast VXLAN.

    About 6 months ago Avaya's VSP platform documentation was available, but their site was/is frustratingly hard to navigate. I just gave it a 10 minute try and gave up. IIRC, the VSP 7200 & 8400 support VXLAN, but it's very early implementation.

    Both Avaya and ALE's switches are Broadcom based, which means they both suffer from the problem of not having the right forwarding pipeline stage ordering/count to put a router interface inside a VXLAN VTEP or inside an SPB I-SID. Avaya solves it with some loopback traces on the board (good luck figuring out specs). ALE solves it by saying "use a loopback cable in a link-agg configuration".

    The VXLAN Routing "yes" for HP gives me pause. I believe a lot of HP's 1U/2U lineup is Trident II based. While ComWare may support the construct, the hardware might not.

    ReplyDelete
    Replies
    1. The information about ALE OmniSwitch is accurate. The OS6900-Q32 and OS6900-X72 both provide VXLAN L2 Gateway in hardware (BCM Trident II). Public documentation is at: http://enterprise.alcatel-lucent.com/assets/documents/OmniSwitch_7_DataCenter_Switching_Guide_Rev-F_EN.pdf or http://enterprise.alcatel-lucent.com/includes/documentlinkPostEloq.cfm?id=22516 . You might want to fast forward to chapter 7, which details what is supported with regards to VXLAN. One should notice that ALE is the only vendor that can provide visibility into the VXLAN overlays with all OS6900/OS10K (VM / VXLAN Snooping) detailed in chapter 8. ALE as also published a whitepaper on the interoperability with VMware NSX (v6.1 was tested) here: http://enterprise.alcatel-lucent.com/includes/documentlinkPostEloq.cfm?id=26782 If you have any questions on that, feel free to reach out to me on Twitter @bennye_hh

      Delete
  11. HP FF 5930 supports VXLAN and OVSDB both.
    A multicast enabled underlay network is not required for HP VXLAN solutions .
    Please refer:
    http://www8.hp.com/h20195/v2/getpdf.aspx/4AA5-7065ENW.pdf?ver=1.0

    ReplyDelete
  12. The ALU/Nuage box is called the 7850 VSG. The data sheet can be found here:
    http://www.nuagenetworks.net/wp-content/uploads/2014/11/NP2013102921EN_V2_NN_7850_VSG_Datasheet1.pdf

    From my memory it supports VXLAN w/EVPN control plane since that's what Nuage uses, not sure about OVSDB. It's Broadcom based so probably doesn't do VXLAN routing.

    EVPN w/VXLAN encap is coming on the QFX switches like the 5100 but isn't available yet.

    ReplyDelete
    Replies
    1. Hi!

      I know what ALU/Nuage box does, but as I cannot get to publicly available documentation, it's not in the table.

      Some vendors might not like this stance. They are always free to change their documentation policy ;)

      Best, Ivan

      Delete
    2. As shown in Ivan's webinar http://www.ipspace.net/Scaling_Overlay_Virtual_Networks the Nuage 7850 VSG does support VXLAN routing, even using the Broadcom Trident II. It does not use OVSDB

      Delete
  13. http://docs.citrix.com/en-us/netscaler/11/networking/vxlans.html

    ReplyDelete
    Replies
    1. Added. Thank you! It's really nice to see more appliance vendors having native VXLAN support!

      Delete
  14. HP 5930 does not support VXLAN Routing. You need a loopback cable to get the exVXLANed packages back into the switch.
    HP 5930 does not support EVPN
    HP 5930 does not support HP´s EVI, i.e. GRE tunnels and IS-IS

    ReplyDelete
  15. Arista does support Multicast VXLAN but not on all platforms specifically not for 7050X series.

    ReplyDelete
  16. To update my own comment. with latest EOS release EOS-4.15.2F Arista now supports Multicast VXLAN for platforms like 7050X, 7250X and 7300X. Haven't tested it though yet.

    ReplyDelete
  17. Juniper just release the 14.1X53-D30 for QFX5100 that include EVPN VxLAN and now they have documentation for it
    http://www.juniper.net/techpubs/en_US/junos14.1/topics/topic-map/sdn-vxlan.html
    (if you can call this mess a documentation ) so you can add QFX5100 and MX to the EVPN section

    ReplyDelete
    Replies
    1. The document you cited contains no mention of EVPN. Still not documented.

      Delete
    2. Better doc: "Juniper Networks EVPN Implementation for Next-Generation Data Center Architectures"
      http://www.juniper.net/assets/us/en/local/pdf/whitepapers/2000606-en.pdf

      Delete
  18. Besides hardware switch. Under Linux, OVS can bridge to VXLAN. A Windows base open source tool UBridge which can be run on most kinds of Windows platform (XP/Vista/Win 7/8/ Win servers etc.) also allows the direct bridge of standalone Windows machine to the VXLAN. Pls refer to following link for detail:

    http://www.techezone.com/question/direct-bridging-of-windows-platform-to-the-vxlan/

    ReplyDelete

You don't have to log in to post a comment, but please do provide your real name/URL. Anonymous comments might get deleted.