Short summary for differently-attentive: proprietary load balancing Brocade uses over ISL trunks in VCS fabric is almost perfect (and way better for high-throughput sessions than what you get with other link aggregation methods).
During the Data Center Fabrics Packet Pushers Podcast we’ve been discussing load balancing across aggregated inter-switch links and Brocade’s claims that its “chip-based balancing” performs better than standard link aggregation group (LAG) load balancing. Ever skeptical, I said all LAG load balancing is chip-based (every vendor does high-speed switching in hardware). I also added that I would be mightily impressed if they’d actually solved intra-flow packet scheduling.
A few days ago Greg (@etherealmind) Ferro, the Packet Pushers host, received a nice e-mail from Michael Schipp containing two slides from a Brocade presentation and “Ivan owes a WOW” PS. I do owe a huge WOW ... but it takes a bit more than just a few slides to impress me (after all, Brook Reams published most of the information contained on those slides a while ago). However, Brook got in touch with me a few days after the podcast was published and provided enough in-depth information to make me a believer (thank you, Brook!).
The first thing Brocade did right (and it should have been standardized and implemented in all switches a long time ago) is automatic trunk discovery: whenever two VDX switches are connected with parallel physical links, those links are automatically trunked. To make use of the advanced load-balancing methods, they also have to be in the same port group (connected to the same chipset), which does reduce the resilience, but if that’s a concern, you can always have two (or more) parallel trunks; TRILL will provide per-MAC-address load balancing across the trunks.
Within each port group, Brocade’s hardware is able to perform per-packet round-robin frame scheduling with guaranteed in-order delivery. It does seem like a magic and it’s not documented anywhere (another painful difference between Brocade and Cisco – Cisco would be more than happy to flaunt its technology wonders), but Brook told me the magic sauce is hidden somewhere within Brocade’s patents and was also kind enough to point me to the most relevant patent.
Based on what’s in that patent (after stripping away all the “we might also be patenting the spread of high-pressure water flows over coffee beans in espresso machine” stuff), it seems that Brocade’s hardware measures link delay and inter-link skew and combines that to schedule the frame transmission in a way that guarantees the frames will always be received in order by the remote switch. They don’t do receiver-side reordering (which is hard), but transmit-side delaying. Very clever solution deserving a huge WOOOOW.
You might wonder how Brocade, a company with historical focus on Fiber Channel, managed to solve one of the tough LAN networking problems almost a decade ago. As you probably know, the networking industry has been in the just-good-enough-to-sell mode for decades. The link aggregation load balancing problem was always way below the pain threshold as a high-speed LAG (port channel) trunk usually carries many flows; doing per-flow (or even per-IP-address) load balancing across a LAG is most often good enough. Storage networking is different: a server servicing hundreds or thousands of users (with at least as many LAN sessions) has only a few storage sessions. Perfect load balancing was thus always a critical component of a SAN network ... and it just happens to be a great solution in LAN environments using iSCSI or NFS storage connectivity.
To learn more about storage networking protocols, including Fiber Channel, iSCSI and NFS, and emerging Data Center technologies including TRILL, Multi-chassis Link Aggregation, FabricPath, FCoE and others watch my Data Center 3.0 for Networking Engineers webinar (buy a recording or yearly subscription).