Category: VXLAN

Even though I wrote about the challenges of routing from VXLAN VNI to VLAN segment on a certain popular chipset a while ago, many engineers obviously still find the topic highly confusing (no surprise there, it is).

Maybe a video is worth a thousand words ;) – I published the part of recent VXLAN webinar where I described the issue in as many details as I could.

Last week I ran the second part of the updated (4-hour) VXLAN webinar. The raw videos are already online and cover these topics:

• VXLAN-related technologies, including encapsulation, IP multicast use, unicast VXLAN, and VXLAN-over-EVPN;
• VXLAN implementations, including Cisco Nexus 1000v, VMware vCNS, VMware NSX, Nuage VSP and Juniper Contrail;
• VXLAN gateways, including Arista, Brocade, Cisco and Juniper;
• Hardware VTEP integration with OVSDB and EVPN;
• VXLAN-based data center fabrics, including Cisco’s ACI.

I’m still getting questions about layer-2 data center interconnect; it seems this particular bad idea isn’t going away any time soon. In the face of that sad reality, let’s revisit what I wrote about layer-2 DCI over VXLAN.

VXLAN hasn’t changed much since the time I explained why it’s not the right technology for long-distance VLANs.

VXLAN is becoming de-facto encapsulation standard for overlay virtual networks (at least according to industry pundits and marketing gurus working for companies with VXLAN-based products) – even Juniper Contrail, which was traditionally a pure MPLS/VPN architecture uses it.

Not so fast – Contrail is using VXLAN packet format to carry MPLS labels between hypervisors and ToR switches.

Most recently launched data center switches use the Trident 2 chipset, and yet we know almost nothing about its capabilities and limitations. It might not work at linerate, it might have L3 lookup challenges when faced with L2 tunnels, there might be other unpleasant surprises… but we don’t know what they are, because you cannot get Broadcom’s documentation unless you work for a vendor who signed an NDA.

Interestingly, the best source of Trident 2 technical information I found so far happens to be the Cisco Live Nexus 9000 Series Switch Architecture presentation (BRKARC-2222). Here are a few tidbits I got from that presentation and Broadcom’s so-called datasheet.

A while ago I wrote “vMotion over VXLAN is stupid and unnecessary” in a comment to a blog post by Duncan Epping, assuming everyone knew the necessary background details. I was wrong (again).

A networking engineer was trying to persuade me of importance of hardware VXLAN VTEPs. We quickly agreed physical-to-virtual gateways are the primary use case, and he tried to illustrate his point by saying “Imagine you have 1000 servers in your data center and you manage to virtualize 80% of them. How will you connect them to the other 200?” to which I replied, “That doesn’t make any sense.” Here’s why.

Brook Reams sent me an interesting tidbit: Brocade is the first vendor that actually shipped a VXLAN VTEP controlled by a VMware NSX controller. It’s amazing to see how Brocade leapfrogged everyone else (they also added tons of other new functionality in NOS releases 4.0 and 4.1).

Gordon sent me a whole list of NSX gateway questions:

• Do you need a virtual gateway for each VXLAN segment or can a gateway be the entry/exit point across multiple VXLAN segments?
• Can you setup multiple gateways and specify which VXLAN segments use each gateway?
• Can you cluster gateways together (Active/Active) or do you setup them up as Active/Standby?

The answers obviously depend on whether you’re deploying NSX for multiple hypervisors or NSX for vSphere. Let’s start with the former.

Got a really interesting question from A. Reader: “When and how does VXLAN use IGMP and PIM in transport (underlay) networks?”

Obviously you need IGMP and PIM in multicast environments only (vCNS 5.x, Nexus 1000V in multicast mode).

The long-promised unicast-only VXLAN has finally shipped with the Nexus 1000V release 4.2(1)SV2(2.1) (there must be some logic behind those numbers, but they all look like madness to me). The new Nexus 1000V release brings two significant VXLAN enhancements: unicast-only mode and MAC distribution mode.

VXLAN, one of the first MAC-over-IP (overlay) virtual networking solutions is definitely a major improvement over traditional VLAN-based virtual networking technologies … but not without its own scalability limitations.

Mark Berly spent plenty of time explaining the in-depth intricacies of VXLAN-to-VLAN gateways during our VXLAN Technical Deep Dive webinar. He’s obviously heavily immersed in this challenge and hits 9+ on the Nerd Meter, so you might have to watch the video a few times to get all the nuances. What can I say – we’ll have fun times in the coming years ;)

Mark Berly, the guest star of my VXLAN Technical Deep Dive webinar focused on VXLAN gateways. Here’s the first part of his presentation, explaining what VXLAN gateways are and where you’d need them.

In a comment to the Firewalls in a Small Private Cloud blog post I wrote “VXLAN is NOT a viable inter-DC solution” and Jason wasn’t exactly happy with my blanket response. I hope Jason got a detailed answer in the VXLAN Technical Deep Dive webinar, here’s a somewhat shorter explanation.