Christoph Jaggi asked me a few questions about using VXLAN with EVPN to build data center fabrics and data center interconnects (including active/active data centers). The German version was published on Inside-IT, here’s the English version.
He started with an obvious one:
What is an active-active data center and why would I want to use an active-active data center?
Numerous organizations have multiple data centers for load sharing or disaster recovery purposes. They could use one of their data centers and have the other(s) as warm or cold standby (active/backup setup) or use all data centers at the same time (active/active).
A friend of mine told me about a “VXLAN is insecure, the sky is falling” presentation from RIPE-77 which claims that you can (under certain circumstances) inject packets into VXLAN virtual networks from the Internet.
Welcome back, Captain Obvious. Anyone looking at the VXLAN packet could immediately figure out that there’s no security in VXLAN. I pointed that out several times in my blog posts and presentations, including Cloud Computing Networking (EuroNOG, September 2011) and NSX Architecture webinar (August 2013).
Some (anti)patterns of network industry are way too predictable: every time there’s a new technology marketers start promoting it as the solution for every problem ever imagined. VXLAN was quickly touted as the solution for long-distance vMotion, and now everyone is telling you how to use VXLAN with EVPN to stretch VLANs across multiple data centers.
Does that make sense? It might… based on your requirements and features available on the devices you use to implement the VXLAN/EVPN fabric. We’ll cover the details in a day-long workshop in Zurich (Switzerland) on December 5th. There are still a few places left, register here.
A lightning talk at the recent RIPE77 conference focused on shortcomings of VXLAN and rise of Geneve.
So what are those perceived shortcomings?
No protocol identifier – a single VXLAN VNI cannot carry more than one payload type. Let me point out that MPLS has the same shortcoming, as does IPsec.
One of my readers sent me a series of questions regarding a new cloud deployment where the cloud implementers want to run VXLAN and EVPN on the hypervisor hosts:
I am currently working on a leaf-and-spine VXLAN+ EVPN PoC. At the same time, the systems team in my company is working on building a Cloudstack platform and are insisting on using VXLAN on the compute node even to the point of using BGP for inter-VXLAN traffic on the nodes.
Using VXLAN (or GRE) encap/decap on the hypervisor hosts is nothing new. That’s how NSX and many OpenStack implementations work.
One of the attendees of my Building Next-Generation Data Center online course tried to figure out whether you can build larger broadcast domains with VXLAN than you could with VLANs. Here’s what he sent me:
I'm trying to understand differences or similarities between VLAN and VXLAN technologies in a view of (*cast) domain limitation.
One of my readers found this Culumus Networks article that explains why you can’t have more than a few hundred VXLAN-based VLAN segments on every port of 48-port Trident-2 data center switch.
Expect to see similar limitations in most other chipsets. There’s a huge gap between millions of segments enabled by 24-bit VXLAN Network Identifier and reality of switching silicon. Most switching hardware is also limited to 4K VLANs.
Andy sent me this question:
I'm currently playing around with BGP & VXLANs and wondering: is there anything preventing from building a virtual IXP with VXLAN? This would be then a large layer 2 network - but why have nobody build this to now, or why do internet exchanges do not provide this?
There was at least one IXP that was running on top of VXLAN. I wanted to do a podcast about it with people who helped them build it in early 2015 but one of them got a gag order.
One of my readers was wondering about the stability and scalability of large layer-2 domains implemented with VXLAN. He wrote:
If common BUM traffic (e.g. ARP) is being handled/localized by the network (e.g. NSX or ACI), and if we are managing what traffic hosts can send with micro-segmentation style filtering blocking broadcast/multicast, are large layer-2 domains still a recipe for disaster?
There are three major (fundamental) problems with large L2 domains:
Cumulus Linux 3.2 shipped with a rudimentary EVPN implementation and everyone got really excited, including smaller ASIC manufacturers that finally got a control plane for their hardware VTEP functionality.
However, while it’s nice to have EVPN support in Cumulus Linux, the claims of its benefits are sometimes greatly exaggerated.
From the moment Cisco and VMware announced VXLAN some networking engineers complained that they'd lose visibility into the end-to-end path. It took a long while, but finally the troubleshooting tools started appearing in VXLAN environment: NVO3 working group defined Fault Managemnet framework for overlay networks and Cisco implemented at least parts of it in recent Nexus OS releases.
Does Cisco ACI use VXLAN inside the fabric or is something else used instead of VXLAN?
ACI uses VXLAN but not in a way that would be (AFAIK) interoperable with any non-Cisco product. While they do use some proprietary tagging bits, the real challenge is the control plane.
I proposed a third option: go with simple VXLAN encapsulation on Nexus 9000 switches. Here’s why:
Do you need VXLAN in your data center or could you continue using traditional bridging? Do layer-2 fabrics make sense or are they a dead end in the evolution of virtual networking?
I tried to provide a few high-level answers in the Introduction to VXLAN video which starts the VXLAN Technical Deep Dive webinar. The public version of the video is now available on ipSpace.net Free Content web site.
One of my readers stumbled upon a 4-year-old blog post explaining the potential implementations of VXLAN hardware gateways, and asked me if that information is still relevant.