Multiple overlay network encapsulations are nothing more than a major inconvenience (and religious wars based on individual bit fields close to meaningless) for anyone trying to support more than one overlay virtual networking technology (just ask F5 or Arista).
The key differentiator between scalable and not-so-very-scalable architectures and technologies is the control plane – the mechanism that maps (at the very minimum) remote VM MAC address into a transport network IP address of the target hypervisor (see A Day in a Life of an Overlaid Virtual Packet for more details).
Overlay virtual networking vendors chose a plethora of solutions to solve this problem, ranging from Ethernet-like dynamic MAC address learning to complex protocols like MP-BGP. Here’s an overview of what they’re doing:
The original VXLAN as implemented by Cisco’s Nexus 1000V, VMware’s vCNS release 5.1, Arista EOS, and F5 BIG-IP TMOS release 11.4 has no control plane. It relies on transport network IP multicast to flood BUM traffic and uses Ethernet-like MAC address learning to build mapping between virtual network MAC address and transport network IP addresses.
Unicast VXLAN as implemented in Cisco’s Nexus 1000V release 4.2(1)SV2(2.1) has something that resembles a control plane. VSM distributes segment-to-VTEP mappings to VEMs to replace IP multicast with headend unicast replication, but the VEMs still use dynamic MAC learning.
VXLAN MAC distribution mode in Nexus 1000V is a proper control plane implementation in which the VSM distributes VM-MAC-to-VTEP-IP information to VEMs. Unfortunately it seems to be based on a proprietary protocol, so it won’t work with hardware gateways from Arista or F5.
Hyper-V Network Virtualization uses PowerShell cmdlets to configure VM-MAC-to-transport-IP mappings, virtual network ARP tables and virtual network IP routing tables. The same cmdlets can be implemented by hardware vendors to configure NVGRE gateways.
Nicira NVP (part of VMware NSX) uses OpenFlow to install forwarding entries in the hypervisor switches and Open vSwitch Database Management Protocol to configure the hypervisor switches. NVP uses OpenFlow to implement L2 forwarding and VM NIC reflexive ACLs (L3 forwarding uses another agent in every hypervisor host).
Midokura Midonet doesn’t have a central controller or control-plane protocols. Midonet agents residing in individual hypervisors use shared database to store control- and data-plane state.
Contrail (now Juniper JunosV Contrail) seems to be using MP-BGP to pass MPLS/VPN information between controllers and XMPP to connect hypervisor switches to the controllers.
IBM SDN-VE (SDN for Virtual Environments) uses a hierarchy of controllers and appliances to implement NVP-like control plane for L2 and L3 forwarding using VXLAN encapsulation. I wasn’t able to figure out what protocols they use from their whitepapers and user guides.
Nuage Networks is using $Something and PLUMgrid is using $SomethingElse. I will tell you what the values of these two variables are when I manage to get product documentation from these vendors. PowerPoint and whitepapers clearly get way more attention in the startup world than something an actual user might find useful when deploying the product.