Category: overlay networks

Meeting Brad Hedlund in person was definitely one of the highlights of my Interop 2013 week. We had an awesome conversation and quickly realized how closely aligned our views of VLANs, overlay networks and virtual appliances are.

Not surprisingly, Brad quickly improved my ideas with a radical proposal: running BGP between the virtual and the physical world.

One of my blogger friends sent me an interesting observation:

After talking to networking vendors I'm inclined to think they are going to focus on a mesh of overlays from the TOR, with possible use of overlays between vswitch and TOR too if desired - drawing analogies to MPLS with ToR a PE and vSwitch a CE. Aside from selling more hardware for this, I'm not drawn towards a solution like this bc it doesn't help with full network virtualization and a network abstraction for VMs.

The whole situation reminds me of the good old SNA and APPN days with networking vendors playing the IBM part of the comedy.

Brad Hedlund wrote another great article, this one explaining the fundamentals of network virtualization. As you'll see, VMware (and everyone else) aims way higher than replacing VLANs with overlay networks. Highly recommended!

Regardless of what the vendors are telling you, it’s hard to get data center disaster recovery right (unless you’re running regular fire drills), and your job usually gets harder due to the intricate (sometimes undocumented) intertwining of physical and virtual worlds. For example, do you know how to get the firewall and load balancer configurations from the failed site implemented in the equipment currently used at disaster recovery site?

Imagine a simple application stack with a few web servers, app servers and two database servers. There’s a firewall in front of the web servers and a load balancer tying all the segments together.

I was asked to do a short virtual networking presentation during this year’s Microsoft NT Conference in Slovenia. Most of the audience were server and virtualization administrators, having anywhere from zero to pretty decent networking knowledge; getting the right balance of basics and interesting features was a struggle.

They told me the end result wasn’t that bad. It’s a bit Microsoft-biased, but applies equally well to VMware (be it vSphere/VXLAN or Open vSwitch/NVP combo).

Hatem Naguib claimed that “the NSX controller cluster is completely out-of-band, and never handles a data packet” when describing VMware NSX Network Virtualization architecture, preemptively avoiding the “flow-based forwarding doesn’t scale” arguments usually triggered by stupidities like this one.

Does that mean there’s no packet punting in the NSX/Open vSwitch world? Not so fast.

VXLAN, one of the first MAC-over-IP (overlay) virtual networking solutions is definitely a major improvement over traditional VLAN-based virtual networking technologies … but not without its own scalability limitations.

In a comment to the Firewalls in a Small Private Cloud blog post I wrote “VXLAN is NOT a viable inter-DC solution” and Jason wasn’t exactly happy with my blanket response. I hope Jason got a detailed answer in the VXLAN Technical Deep Dive webinar, here’s a somewhat shorter explanation.

Kaage added a great comment to my Virtual Firewall Taxonomy post:

And many of physical firewalls can be virtualized. One physical firewall can have multiple virtual firewalls inside. They all have their own routing table, rule base and management interface.

He’s absolutely right, but there’s a huge difference between security contexts (to use the ASA terminology) and firewalls running in VMs.

Mrs. Y, the network security princess, sent me an interesting design challenge:

We’re building a private cloud and I'm pushing for keeping east/west traffic inside the cloud. What are your opinions on the pros/cons of keeping east/west traffic in the cloud vs. letting it exit for security/routing?

Short answer: it depends.

Dumlu Timuralp (@dumlutimuralp) sent me an excellent question:

I always get confused when thinking about IP multicast traffic over VXLAN tunnels. Since VXLAN already uses a Multicast Group for layer-2 flooding, I guess all VTEPs would have to receive the multicast traffic from a VM, as it appears as L2 multicast. Am I missing something?

Short answer: no, you’re absolutely right. IP multicast over VXLAN is clearly suboptimal.

Arista is launching a new product line today shrouded in mists of SDN and cloud buzzwords: the 7150 series top-of-rack switches. As expected, the switches offer up to 64 10GE ports with wire speed L2 and L3 forwarding and 400 nanosecond(!) latency.

Also expected from Arista: unexpected creativity. Instead of providing a 40GE port on the switch that can be split into four 10GE ports with a breakout cable (like everyone else is doing), these switches group four physical 10GE SFP+ ports into a native 40GE (not 4x10GE LAG) interface.

But wait, there’s more...

Almost everyone agrees the current way of implementing virtual networks with dumb hypervisor switches and top-of-rack kludges (including Edge Virtual Bridging – EVB or 802.1Qbg – and 802.1BR) doesn’t scale. Most people working in the field (with the notable exception of some hardware vendors busy protecting their turfs in the NVO3 IETF working group) also agree virtual networks running as applications on top of IP fabric are the only reasonable way to go ... but that’s all they currently agree upon.

When VXLAN came out a year ago, a lot of us looked at the packet format and wondered why Cisco and VMware decided to use UDP instead of more commonly used GRE. One explanation was evident: UDP port numbers give you more entropy that you can use in 5-tuple-based load balancing. The other explanation looked even more promising: VXLAN and OTV use very similar packet format, so the hardware already doing OTV encapsulation (Nexus 7000) could be used to do VXLAN termination. Boy have we been suckered.

Update 2015-07-12: NX-OS 7.2.0 supports OTV encapsulation with VXLAN-like headers on F3 linecards. See OTV UDP Encapsulation for more details (HT: Nik Geyer).

Aldrin Isaac made a great comment to my Could MPLS-over-IP replace VXLAN? article:

As far as I understand, VXLAN, NVGRE and any tunneling protocol that use global ID in the data plane cannot support PVLAN functionality.

He’s absolutely right, but you shouldn’t try to shoehorn VXLAN into existing deployment models. To understand why that doesn’t make sense, we have to focus on the typical cloud application architectures.