Category: VXLAN

Got a really interesting question from A. Reader: “When and how does VXLAN use IGMP and PIM in transport (underlay) networks?”

Obviously you need IGMP and PIM in multicast environments only (vCNS 5.x, Nexus 1000V in multicast mode).

The long-promised unicast-only VXLAN has finally shipped with the Nexus 1000V release 4.2(1)SV2(2.1) (there must be some logic behind those numbers, but they all look like madness to me). The new Nexus 1000V release brings two significant VXLAN enhancements: unicast-only mode and MAC distribution mode.

VXLAN, one of the first MAC-over-IP (overlay) virtual networking solutions is definitely a major improvement over traditional VLAN-based virtual networking technologies … but not without its own scalability limitations.

Mark Berly spent plenty of time explaining the in-depth intricacies of VXLAN-to-VLAN gateways during our VXLAN Technical Deep Dive webinar. He’s obviously heavily immersed in this challenge and hits 9+ on the Nerd Meter, so you might have to watch the video a few times to get all the nuances. What can I say – we’ll have fun times in the coming years ;)

Mark Berly, the guest star of my VXLAN Technical Deep Dive webinar focused on VXLAN gateways. Here’s the first part of his presentation, explaining what VXLAN gateways are and where you’d need them.

In a comment to the Firewalls in a Small Private Cloud blog post I wrote “VXLAN is NOT a viable inter-DC solution” and Jason wasn’t exactly happy with my blanket response. I hope Jason got a detailed answer in the VXLAN Technical Deep Dive webinar, here’s a somewhat shorter explanation.

Kaage added a great comment to my Virtual Firewall Taxonomy post:

And many of physical firewalls can be virtualized. One physical firewall can have multiple virtual firewalls inside. They all have their own routing table, rule base and management interface.

He’s absolutely right, but there’s a huge difference between security contexts (to use the ASA terminology) and firewalls running in VMs.

Mrs. Y, the network security princess, sent me an interesting design challenge:

We’re building a private cloud and I'm pushing for keeping east/west traffic inside the cloud. What are your opinions on the pros/cons of keeping east/west traffic in the cloud vs. letting it exit for security/routing?

Short answer: it depends.

Dumlu Timuralp (@dumlutimuralp) sent me an excellent question:

I always get confused when thinking about IP multicast traffic over VXLAN tunnels. Since VXLAN already uses a Multicast Group for layer-2 flooding, I guess all VTEPs would have to receive the multicast traffic from a VM, as it appears as L2 multicast. Am I missing something?

Short answer: no, you’re absolutely right. IP multicast over VXLAN is clearly suboptimal.

Arista is launching a new product line today shrouded in mists of SDN and cloud buzzwords: the 7150 series top-of-rack switches. As expected, the switches offer up to 64 10GE ports with wire speed L2 and L3 forwarding and 400 nanosecond(!) latency.

Also expected from Arista: unexpected creativity. Instead of providing a 40GE port on the switch that can be split into four 10GE ports with a breakout cable (like everyone else is doing), these switches group four physical 10GE SFP+ ports into a native 40GE (not 4x10GE LAG) interface.

But wait, there’s more...

When VXLAN came out a year ago, a lot of us looked at the packet format and wondered why Cisco and VMware decided to use UDP instead of more commonly used GRE. One explanation was evident: UDP port numbers give you more entropy that you can use in 5-tuple-based load balancing. The other explanation looked even more promising: VXLAN and OTV use very similar packet format, so the hardware already doing OTV encapsulation (Nexus 7000) could be used to do VXLAN termination. Boy have we been suckered.

Update 2015-07-12: NX-OS 7.2.0 supports OTV encapsulation with VXLAN-like headers on F3 linecards. See OTV UDP Encapsulation for more details (HT: Nik Geyer).

Aldrin Isaac made a great comment to my Could MPLS-over-IP replace VXLAN? article:

As far as I understand, VXLAN, NVGRE and any tunneling protocol that use global ID in the data plane cannot support PVLAN functionality.

He’s absolutely right, but you shouldn’t try to shoehorn VXLAN into existing deployment models. To understand why that doesn’t make sense, we have to focus on the typical cloud application architectures.

A lot of engineers are concerned with what seems to be frivolous creation of new encapsulation formats supporting virtual networks. While STT makes technical sense (it allows soft switches to use existing NIC TCP offload functionality), it’s harder to figure out the benefits of VXLAN and NVGRE. Scott Lowe wrote a great blog post recently where he asked a very valid question: “Couldn’t we use MPLS over GRE or IP?” We could, but we wouldn’t gain anything by doing that.

I usually use the “Nicira is Skype of virtual networking” analogy when describing the differences between Nicira’s NVP and traditional VLAN-based implementations. Cade Metz liked it so much he used it in his What Is a Virtual Network? It’s Not What You Think It Is article, so I guess a blog post is long overdue.

Before going into more details, you might want to browse through my Cloud Networking Scalability presentation (or watch its recording) – the crucial slide is this one:

VMware has a fantastic-looking cloud provisioning tool – vCloud director. It allows cloud tenants to deploy their VMs and create new virtual networks with a click of a mouse (the underlying network has to provide a range of VLANs, or you could use VXLAN or vCDNI to implement the virtual segments).

Needless to say, when engineers not familiar with the networking intricacies create point-and-click application stacks without firewalls and load balancers, you get some interesting designs.