Yesterday I described how the lack of LACP support in VMware’s vSwitch and vDS can limit the load balancing options offered by the upstream switches. The situation gets totally out-of-hand when you connect an ESX server with two uplinks to two (or more) switches that are part of a Multi-chassis Link Aggregation (MLAG) cluster.
Let’s expand the small network described in the previous post a bit, adding a second ESX server and another switch. Both ESX servers are connected to both switches (resulting in a fully redundant design) and the switches have been configured as a MLAG cluster (using VSS with Catalyst 6500, vPC with Nexus 7000 or Nexus 5000, or IRF with the HP switches). Link aggregation is not used between the physical switches and ESX servers due to lack of LACP support in ESX.
The physical switches are unaware of the physical connectivity the ESX servers have. Assuming that vSwitches use per-VM load balancing and each VM is pinned to one of the uplinks, source MAC address learning in the physical switches produces the following logical topology:
Each VM appears to be single-homed to one of the switches. The traffic between VM A and VM C is thus forwarded locally by the left-hand switch; the traffic between VM A and VM D has to traverse the inter-switch link because neither switch knows VM D can also be reached by a shorter path.
In a Multi-chassis Link Aggregation scenario it’s thus almost mandatory to configure static port channel on the switches to which the vSphere servers are connected, otherwise you risk overloading the inter-switch link as the traffic between adjacent ESX servers is needlessly sent across that link. When doing that, you (probably) have to configure IP-hash-based load balancing in vSwitch (more information about the vSwitch-side implications if the NICs in ESX are configured in active/standby configuration).
It's much better (at least from Cisco’s perspective) to use Nexus 1000V – it supports LACP, ESX servers start behaving like access-layer switches and the traffic flow always remains optimal (at least within the boundaries of hot-potato switching).