One of my readers has customers that already experienced performance challenges with Tomahawk-based data center switches. He sent me an email along these lines:
My customers are concerned about buffer performance for packets that are 200 bytes and under. MORE IMPORTANTLY, a customer informed me that there were performance issues when running 4x25GE connections when one group of ports speaks to another group.
Reading the report Mellanox published not so long ago it seems there really is something fishy going on with Tomahawk.
Let’s be realistic: every chipset has limitations. Every silicon vendor has to make tradeoffs. Every vendor loves to find a chink in another vendor’s product and tries to present it as a glaring abyss.
However, there are vendors that are pretty open about their chipset architectures. Well, make it A Vendor – Cisco has some amazing “a day in the life of a packet” Cisco Live presentations that anyone with an email address can watch (if there’s someone else as open as Cisco is about their internal architectures, please post a comment).
Then there are vendors that document the realistic boundaries of their architectures. For example, when Cisco was selling oversubscribed 10GE linecards for Nexus 7000 they documented which ports share the forwarding bandwidth. Arista was pretty open about the forwarding performance of their 7300 switches (so it was easy to deduce they are not linerate at small packet sizes). Brocade documented their ASIC limitations in regards to link aggregation.
So what’s the problem with Broadcom chipsets? We don’t know what the limitations are because they’re hiding the information, and everyone who does know what’s really going on can’t talk about it.
Well, the real problem is that Broadcom doesn’t have to care – every major data center switching vendor (apart from Brocade, a few Arista and Cisco switches, and Juniper QFX10K) is using some Broadcom chipset.
Before someone starts listing a dozen reasons why hiding that information makes sense, let me point out that Intel publishes all of their documentation.
The only way to get any information on how the stuff that’s sitting inside most of data center switches these days really works is to try to reverse-engineer tidbits from various vendor presentations or their configuration guidelines, or rely on Broadcom competitors spilling the beans disguised as a test report. And yet some people call this state of affairs progress.
For a more articulate take on the same topic, read the excellent blog post by Carlos Cardenas, and if you want to know more about implementing a networking operating system, register for my online course with Russ White as the guest speaker.