Virtual (soft) switches present in almost every hypervisor significantly reduce the performance of high-bandwidth virtual machines (measurements done by Cisco a while ago indicate you could get up to 38% more throughput if you tie VMs directly to hardware NICs), but as I argued in my “Soft Switching Might Not Scale, But We Need It” post, we need hypervisor switches to isolate the virtual machines from the vagaries of the physical NICs.
Engineering gurus from Cisco and VMware have yet again proven me wrong – you can combine VMDirectPath and vMotion if you use VM-FEX.
This is (approximately) how that marvel of engineering works (and you’ll find more details in this presentation):
- You have to configure VM-FEX (which means that you can only use this trick if you have an UCS system with Palo chipset in the server blades).
- Palo chipset emulates the registers and data structures used by the VMXNET3 paravirtualized device driver (and most VMs use VMXNET3 today due to its performance benefits). You can thus link a VM with VMXNET3 device driver directly to the physical hardware presented to the server by the Palo chipset (using VMDirectPath, for example).
Cisco was using VMDirectPath in the VM-FEX performance measurements; in most VM-FEX deployments you’d use the passthrough VEM to enable vMotion of the VMs using VM-FEX.
- vSphere 5 introduced support for vMotion with VMDirectPath for VM-FEX NICs. This enhancement is crucial as it allows a VM using VM-FEX NIC without a VEM to be vMotioned to another host.
The trick VMware’s engineers used is very simple (conceptually, but I’m positive there are numerous highly convoluted implementation details): once you get a request to vMotion a VM, you freeze the VM, copy physical registers of the VM-FEX VIC to the data structures used by the hypervisor kernel implementation of VMXNET3 device, disconnect the VM from the physical hardware, and allow it to continue working through the virtual VMXNET3 device and VEM. Once the VM is moved to another ESX host, the contents of the VMXNET3 virtual device registers get copied to the physical NIC, and the VM yet again regains full access to the physical hardware.
Was it all just an alphabet soup?
Check out my virtualization webinars – they will help you get a decent foothold in the brave new world of server and network virtualization.