Category: fabric

Enterasys implemented optimal layer-3 forwarding with an interesting trick: they support VRRP like any other switch vendor, but allow you to make all members of a VRRP group active forwarders regardless of their status.

Apart from a slightly more synchronized behavior, their implementation doesn’t differ much from Arista’s Virtual ARP, and thus shares the same design and deployment caveats.

For more information, watch the Fabric Routing video from the Enterasys Robust Data Center Interconnect Solutions webinar.

During the recent Data Center Fabrics Update webinar Dan Backman from Plexxi explained how their innovative use of CWDM technology and controller-assisted forwarding simplifies deployment and growth of reasonably-sized data center fabrics.

I would highly recommend that you watch the video – the start is a bit short on details, but he does cover all the juicy aspects later on.

In the Optimal L3 Forwarding with VARP and Active/Active VRRP blog post I made a remark along the lines of “Things might get nasty [in Arista EOS Virtual ARP world] if you have configuration mismatches”, resulting in a lengthy and amazingly insightful email exchange with Lincoln Dale during which we ventured deeper and deeper down the Virtual ARP (VARP) rabbit hole. Here’s what I learned during out trip:

The Optimal L3 Forwarding with VARP/VRRP post generated numerous comments, ranging from technical questions about VARP (more about that in a few days) to remarks along the lines of “you can do that with X” or “vendor Y supports Z, which does the same thing.” It seems I’ve opened yet another can of worms, let’s try to tame and sort them.

I’ve blogged about the need for optimal L3 forwarding across the whole data center in 2012 when I introduced it as one of the interesting requirements in Data Center Fabrics webinar. Years later, the concept became one of the cornerstones of modern EVPN fabrics, but there are still only a few companies that can deliver this functionality in a more traditional environment.

A few days after I published the Interop Product Launch Craze post, Jason Edelman told me HP claims they have running TRILL implementation. Time to read their release notes.

Results: No mention of TRILL in latest release notes for 12500, 9500 or 58xx. 5900 switches support TRILL, EVB and FCoE since release 2207 (January 2013).

More about changes in the data center switching market in the Data Center Fabrics Update webinar. Now I have to catch the next plane on the way home.

In the last part of Clos Fabrics Explained webinar Brad Hedlund described how you can use Dell Fabric Manager to plan, design, configure and operate leaf-and-spine Clos fabric built with Dell Force10 switches. Should we call Dell Fabric Manager an SDN solution? Who cares, it sure is useful ;)

Plexxi has a really interesting data center fabric solution that combines CWDM optics with L2+L3 switching. They briefed me on their product just before their public launch; I like their approach, particularly the combination of robust traditional forwarding with controller-based network optimization that you can influence from the outside, but somehow I never quite found the time to blog about them … although I did manage to solve the hard part of the problem: write a Perl script that generates Graphviz graph description to generate schematics of their CWDM inter-switch links.

As expected, we’ve experienced a product launch craze just prior to Interop Las Vegas. I try to avoid marketing announcements, but the blogosphere exploded in hard-to-ignore posts ... and as always, it was great fun separating marketing fluff from reality. Here’s a grumpy take on the above-mentioned press releases.

One of the attendees of the ProgrammableFlow webinar sent me an interesting observation:

Though there is separate control plane and separate data plane, it appears that there is crossover from one to the other. Consider the scenario when flow tables are not programmed and so the packets will be punted by the ingress switch to PFC. The PFC will then forward these packets to the egress switch so that the initial packets are not dropped. So in some sense: we are seeing packet traversing the boundaries of typical data-plane and control-plane and vice-versa.

He’s absolutely right, and if the above description reminds you of fast and process switching you’re spot on. There really is nothing new under the sun.

Once you get rid of spanning tree and associated kludges (not too hard in OpenFlow-based networks), BUM flooding becomes your biggest enemy. NEC’s engineers implemented some interesting features in the ProgrammableFlow switches and controllers: rate-limiting of unknown unicast frames, flooding control, and ARP snooping (if only they’d go for ARP proxy).

OpenFlow is not exactly known for its quality-of-service features (hint: there are none), but as I described in the ProgrammableFlow Technical Deep Dive webinar NEC implemented numerous OpenFlow extensions in their edge switches and the ProgrammableFlow controller to give you a robust set of QoS features.

Smaller Clos fabrics are built with two layers of switches: leaf and spine switches. The oversubscription ratio you want to achieve dictates the number of uplinks on the leaf switch, which in turn dictates the maximum number of spine switches and thus the fabric size.

You have to use multi-stage Clos architecture if you want to build bigger fabrics; Brad Hedlund described a sample fabric with over 24.000 server-facing ports in the Clos Fabrics Explained webinar.

Matt Thompson provided a really good answer to the “what’s acceptable oversubscription ratio in a ToR switch” when he wrote “I’m expecting a ‘how long is a piece of string’ answer” (note: do watch the BBC video answering that one).

There’s the 3:1 rule-of-thumb recipe, with a more realistic answer being “it depends”. Now let’s see if we can go beyond that without a deep dive into scholastic waters.

I had an interesting conversation with Doug Hanks (@douglashanksjr) about the need for intra-spine links in leaf-and-spine fabric designs. You clearly don’t need links between spine switches when every leaf node (switch or router/firewall/load balancer) is connected to all spine switches ... but what happens when one of the leaf-to-spine links fails? Will other leaf switches know that they have to avoid the spine switch with the failed link?