Category: VXLAN

Andy sent me this question:

I'm currently playing around with BGP & VXLANs and wondering: is there anything preventing from building a virtual IXP with VXLAN? This would be then a large layer 2 network - but why have nobody build this to now, or why do internet exchanges do not provide this?

There was at least one IXP that was running on top of VXLAN. I wanted to do a podcast about it with people who helped them build it in early 2015 but one of them got a gag order.

One of my readers was wondering about the stability and scalability of large layer-2 domains implemented with VXLAN. He wrote:

If common BUM traffic (e.g. ARP) is being handled/localized by the network (e.g. NSX or ACI), and if we are managing what traffic hosts can send with micro-segmentation style filtering blocking broadcast/multicast, are large layer-2 domains still a recipe for disaster?

There are three major (fundamental) problems with large L2 domains:

Cumulus Linux 3.2 shipped with a rudimentary EVPN implementation and everyone got really excited, including smaller ASIC manufacturers that finally got a control plane for their hardware VTEP functionality.

However, while it’s nice to have EVPN support in Cumulus Linux, the claims of its benefits are sometimes greatly exaggerated.

From the moment Cisco and VMware announced VXLAN some networking engineers complained that they'd lose visibility into the end-to-end path. It took a long while, but finally the troubleshooting tools started appearing in VXLAN environment: NVO3 working group defined Fault Managemnet framework for overlay networks and Cisco implemented at least parts of it in recent Nexus OS releases.

You'll find more details in Software Gone Wild Episode 69 recorded with Lukas Krattiger in November 2016 (you can also watch VXLAN Technical Deep Dive webinar to learn more about VXLAN).

I got a long list of VXLAN-related questions from one of my subscribers. It started with an easy one:

Does Cisco ACI use VXLAN inside the fabric or is something else used instead of VXLAN?

ACI uses VXLAN but not in a way that would be (AFAIK) interoperable with any non-Cisco product. While they do use some proprietary tagging bits, the real challenge is the control plane.

One of my friends plans to replace existing FabricPath data center infrastructure, and asked whether it would make sense to stay with FabricPath (using the new Nexus 5600 switches) or migrate to ACI.

I proposed a third option: go with simple VXLAN encapsulation on Nexus 9000 switches. Here’s why:

Do you need VXLAN in your data center or could you continue using traditional bridging? Do layer-2 fabrics make sense or are they a dead end in the evolution of virtual networking?

I tried to provide a few high-level answers in the Introduction to VXLAN video which starts the VXLAN Technical Deep Dive webinar. The public version of the video is now available on ipSpace.net Free Content web site.

One of my readers stumbled upon blog post from 2011 explaining the potential implementations of VXLAN hardware gateways, and asked me if that information is still relevant.

I knew that I’d included tons of information in the Data Center Fabrics and VXLAN Deep Dive webinars, but couldn’t find anything on the web, so I decided to fix that in 2015.

Every now and then someone actually looks at the VXLAN packet format and eventually figures out that VXLAN encapsulation doesn’t provide any intrinsic security.

TL&DR Summary: That’s old news, the sky is not falling, and deploying VXLAN won’t make your network less secure than traditional VLAN- or MPLS-based networks.

Even though I wrote about the challenges of routing from VXLAN VNI to VLAN segment on a certain popular chipset a while ago, many engineers obviously still find the topic highly confusing (no surprise there, it is).

Maybe a video is worth a thousand words ;) – I published the part of recent VXLAN webinar where I described the issue in as many details as I could.

Last week I ran the second part of the updated (4-hour) VXLAN webinar. The raw videos are already online and cover these topics:

• VXLAN-related technologies, including encapsulation, IP multicast use, unicast VXLAN, and VXLAN-over-EVPN;
• VXLAN implementations, including Cisco Nexus 1000v, VMware vCNS, VMware NSX, Nuage VSP and Juniper Contrail;
• VXLAN gateways, including Arista, Brocade, Cisco and Juniper;
• Hardware VTEP integration with OVSDB and EVPN;
• VXLAN-based data center fabrics, including Cisco’s ACI.

I’m still getting questions about layer-2 data center interconnect; it seems this particular bad idea isn’t going away any time soon. In the face of that sad reality, let’s revisit what I wrote about layer-2 DCI over VXLAN.

VXLAN hasn’t changed much since the time I explained why it’s not the right technology for long-distance VLANs.

VXLAN is becoming de-facto encapsulation standard for overlay virtual networks (at least according to industry pundits and marketing gurus working for companies with VXLAN-based products) – even Juniper Contrail, which was traditionally a pure MPLS/VPN architecture uses it.

Not so fast – Contrail is using VXLAN packet format to carry MPLS labels between hypervisors and ToR switches.

Most recently launched data center switches use the Trident 2 chipset, and yet we know almost nothing about its capabilities and limitations. It might not work at linerate, it might have L3 lookup challenges when faced with L2 tunnels, there might be other unpleasant surprises… but we don’t know what they are, because you cannot get Broadcom’s documentation unless you work for a vendor who signed an NDA.

Interestingly, the best source of Trident 2 technical information I found so far happens to be the Cisco Live Nexus 9000 Series Switch Architecture presentation (BRKARC-2222). Here are a few tidbits I got from that presentation and Broadcom’s so-called datasheet.

A while ago I wrote “vMotion over VXLAN is stupid and unnecessary” in a comment to a blog post by Duncan Epping, assuming everyone knew the necessary background details. I was wrong (again).