Turn a switch into a hub … the Microsoft way
If you’ve ever tried to get advanced Cisco certifications, you’ve probably encountered questions dealing with the mismatch between the end device ARP timeouts and the L2 switch CAM (MAC address cache) timeouts. If you’re still wondering what the underlying problem is (it took me a while to figure it out), read the Unicast Flooding in Switched Campus Networks document from Cisco.
In all scenarios, traffic sent to unknown unicast MAC address causes layer-2 flooding, which can significantly reduce switch performance. Microsoft took this problem to a completely new level with its Network Load Balancing implementation: Windows servers send ARP replies containing MAC address X from MAC address Y, causing all the traffic toward the servers to be flooded – effectively turning an Ethernet switch into a hub.
I ran into this recently with folks complaining about poor voice quality. As it turns out, the CPUs on our 6500s were getting pegged. We tracked it down to a couple "clustered" hosts running Microsoft NLB. We eventually moved them onto a DNS-based load balancing solution off our GSLB which met their needs.
If you read the protocol design on Microsoft Technet, it is truly written by application developers. However, if you absolutely MUST run NLB in your network, definitely go with the multicast option with IGMP snooping to handle any flooding issues. Details here:
In unicast mode, the same MAC address is used for ALL cluster members (who are now of course unable to communicate with one another). The alternative is to use multicast mode - which sounds just peachy, but on a single LAN segment there are no membership requests to snoop so the data floods out every port just like in the unicast model.
Last time I came across this a few years back, my solution was the same as Dragan's - isolation by segmentation. *sigh*