A while ago I wrote “vMotion over VXLAN is stupid and unnecessary” in a comment to a blog post by Duncan Epping, assuming everyone knew the necessary background details. I was wrong (again).
Trying to understand my reasoning Jon sent me a very nice question:
vMotion is an exchange of data between hypervisor kernels, VXLAN is a VM networking solution. I get that, but in one of your videos you say VXLAN is a solution to meet the L2 adjacency requirements of vMotion.
If you design a L3 IP transport network ie. L3 from access to aggregation but you wanted to use vMotion then how could you do that unless you used an overlay technoology such as VXLAN to extend the vlan across the underlying IP network.
My somewhat imprecise claims often get me in trouble (this wouldn’t be the first time), let me try to straighten things out.
- L2 adjacency between source and target hypervisor hosts for the port group in which the VM resides. Without the L2 adjacency you cannot move a live IP address and retain all sessions (solutions like Enterasys’ host routing are an alternative if you don’t mind longer traffic interruptions caused by routing protocol convergence time);
- IP connectivity between vmkernel interface of hypervisor hosts (vMotion uses TCP to transport data between hypervisors). VMware always claimed that you need L2 connectivity between hypervisor hosts, and that vMotion between hosts residing in multiple subnets is unsupported (supposedly it became supported last year), but it always worked.
In other words, when you move a VM, its Ethernet NIC must reside in the same L2 segment after the move, but the vmkernel interfaces of the source and target hypervisor hosts can be in different subnets.
You can implement the VM NIC must be in the same L2 segment requirement with VLANs (which require end-to-end L2 connectivity) or VXLAN (which emulates L2 segments across L3 infrastructure).
- VMware Networking Deep Dive
- Overlay Virtual Networking (now extended with detailed packet flows)
- VXLAN Technical Deep Dive