In August 2011, when NVGRE draft appeared mere days after VXLAN was launched, I dismissed it as “more of the same, different encapsulation, vague control plane”. Boy was I wrong … and pleasantly surprised when I figured out one of the major virtualization vendors actually did the right thing.
TL;DR Summary: Hyper-V Network Virtualization is a layer-3 virtual networking solution with centralized (orchestration system based) control plane. Its scaling properties are thus way better than VXLAN’s (or Nicira’s … unless they implemented L3 forwarding since the last time we spoke).
From the functional block standpoint, Hyper-V Network Virtualization looks very similar to VXLAN. The virtual switch (vmSwitch) embedded in Hyper-V does simple layer-2 switching, and the Windows Network Virtualization (HNV) module, the core building block of the solution, is inserted between an internal VLAN (virtual subnet; VSID) and the physical NIC.
Interesting trivia: HNV is implemented as a loadable NDIS filter (for the physical NIC) and inserted between the vmSwitch and the NIC teaming driver.
The major difference between VXLAN and HNV module is their internal functionality:
- HNV is a full-blown layer-3 switch – it doesn’t do bridging at all; even packets forwarded within a single virtual network (VSID) are forwarded based on their destination IP address.
- HNV is not relying on flooding and dynamic learning to get the VM-MAC-to-VTEP mappings; all the forwarding information is loaded into the HNV module through PowerShell cmdlets.
Like OpenFlow-based solutions (example: Nicira’s NVP), Hyper-V Network Virtualization relies on a central controller (or orchestration system) to get the mappings between VM MAC and hypervisor IP address and VM IP and VM MAC address. It uses those mapping for deterministic packet forwarding (no unicast flooding) and ARP replies (every HNV module is an ARP proxy).
Net result: you don’t need IP multicast in the transport network (unless you need IP multicast within the virtual network – more about that in a follow-up post), and there’s zero flooding, making HNV way more scalable than any other enterprise solution available today.
How well will it scale?
From the scalability perspective, Hyper-V Network Virtualization architecture seems to be pretty close to Amazon’s VPC. The scalability ranking of major virtual network solutions (based on my current understanding of how they work) would thus be:
- Amazon VPC (pure layer-3 IP-over-IP solution)
- Hyper-V Network Virtualization (almost layer-3 solution using MAC-over- GRE encapsulation)
- Nicira’s NVP (layer-2-over-STT/GRE solution with central control plane)
- VXLAN (layer-2-over-IP solution with no control plane)
- VLAN-based solutions
Breaking with the past bad practices
And now for a few caveats inherent in the (pretty optimal) Hyper-V Network Virtualization architecture:
- Since the HNV module performs L3-based forwarding, you cannot run non-IP protocols in the virtual network. However, HNV already supports IPv4 and IPv6, both within the overlay virtual network, and in the transport network. Let me repeat this: Microsoft is the only major virtualization vendor that has shipping IPv6 virtual networking implementation.
- You cannot rely on dirty tricks (that should never have appeared in the first place) like clusters with IP address sharing implemented with ARP spoofing.
I’m positive that the lack of support for dirty layer-2 tricks will upset a few people using them, but it’s evident Microsoft got sick and tired of the bad practice of supporting kludges and decided to boldly go where no enterprise virtualization vendors has dared to go before.
Hyper-V: Boldly going toward the Amazon nebula (image source)
A huge Thank you!
Matthias Backhausen was the first one to alert me to the fact that there’s more to NVGRE than what’s in the IETF draft.
There’s plenty of high-level Hyper-V/HNV documentation available online (see the More details section below) but the intricate details are still somewhat under-documented.
However, my long-time friend Miha Kralj (we know each other since the days when Lotus Notes was considered a revolutionary product) introduced me to CJ Williams and his team (Bob Combs and Praveen Balasubramanian) graciously answered literally dozens of my questions.
A huge Thank you to all of you!
- Network Virtualization technical details
- Hyper-V Network Virtualization Gateway Architectural Guide
- NVGRE: Network Virtualization using Generic Routing Encapsulation
- Step-by-Step: Hyper-V Network Virtualization
For step-by-step hands-on description, read Demystifying Windows Server 2012 Hyper-V 3.0 Network Virtualization series by Luka Manojlović: