Category: data center

Plexxi has a really interesting data center fabric solution that combines CWDM optics with L2+L3 switching. They briefed me on their product just before their public launch; I like their approach, particularly the combination of robust traditional forwarding with controller-based network optimization that you can influence from the outside, but somehow I never quite found the time to blog about them … although I did manage to solve the hard part of the problem: write a Perl script that generates Graphviz graph description to generate schematics of their CWDM inter-switch links.

As expected, we’ve experienced a product launch craze just prior to Interop Las Vegas. I try to avoid marketing announcements, but the blogosphere exploded in hard-to-ignore posts ... and as always, it was great fun separating marketing fluff from reality. Here’s a grumpy take on the above-mentioned press releases.

Yesterday I wrote that it’s pretty easy to develop a VLAN provisioning application (integrating it with vCenter or System Center earns you bonus points, but even that’s not too hard), so based on the frequent “I hate using CLI to provision VLANs” rants you might wonder where all the startups developing those applications are. Simple answer: there’s no reasonably-sized market. How would I know that? We’ve been there.

I love(d) listening to the Packet Pushers podcast and came to expect the following rant in every SDN-focused episode: “I’m sick and tired of using CLI to manually provision VLANs”. Sure, we’re all in the same boat, but did you ever do something to get rid of that problem?

Chris Marget recently asked a really interesting question:

I’ve encountered an environment where the iSCSI networks are built just like FC networks: Multipathing software in use on servers and storage, switches dedicated to “SAN A” and “SAN B” VLANs, and full isolation of paths (redundant paths) between server and storage. I understand creating a dedicated iSCSI VLAN, but why would you need two? Isn’t the whole thing running on top of TCP? Am I missing something?

Well, it actually makes sense in some mission-critical environments.

One of the attendees of the ProgrammableFlow webinar sent me an interesting observation:

Though there is separate control plane and separate data plane, it appears that there is crossover from one to the other. Consider the scenario when flow tables are not programmed and so the packets will be punted by the ingress switch to PFC. The PFC will then forward these packets to the egress switch so that the initial packets are not dropped. So in some sense: we are seeing packet traversing the boundaries of typical data-plane and control-plane and vice-versa.

He’s absolutely right, and if the above description reminds you of fast and process switching you’re spot on. There really is nothing new under the sun.

Once you get rid of spanning tree and associated kludges (not too hard in OpenFlow-based networks), BUM flooding becomes your biggest enemy. NEC’s engineers implemented some interesting features in the ProgrammableFlow switches and controllers: rate-limiting of unknown unicast frames, flooding control, and ARP snooping (if only they’d go for ARP proxy).

Smaller Clos fabrics are built with two layers of switches: leaf and spine switches. The oversubscription ratio you want to achieve dictates the number of uplinks on the leaf switch, which in turn dictates the maximum number of spine switches and thus the fabric size.

You have to use multi-stage Clos architecture if you want to build bigger fabrics; Brad Hedlund described a sample fabric with over 24.000 server-facing ports in the Clos Fabrics Explained webinar.

Another day, another interesting Expert Express engagement, another stretched layer-2 design solving the usual requirement: “We need inter-DC VM mobility.”

The usual question: “And why would you want to vMotion a VM between data centers?” with a refreshing answer: “Oh, no, that would not work for us.”

A while ago I got an interesting question:

Let's say that due to circumstances outside of your control, you must have stretched data center subnets... What is the best method to get these subnets into OSPF? Should they share a common area at each data center or should each data center utilize a separate area for the same subnet?

Assuming someone hasn’t sprinkled the application willy-nilly across the two data centers, it’s best if the data center edge routers advertise subnets used by the applications as type-2 external routes, ensuring one data center is always the primary entry point for a specific subnet. Getting the same results with BGP routing in Internet is a much tougher challenge.

The ProgrammableFlow Principles of Operations section of the ProgrammableFlow Technical Deep Dive webinar generated tens of questions from the audience – it took us almost 20 minutes to answer all of them (note: you might watch the answers after watching the section that triggered the questions).

The second part of ProgrammableFlow Technical Deep Dive webinar describes ProgrammableFlow principles – physical networks (control, data, HA cluster, management), edge and core switches, topology discovery and end-to-end routing and path setup procedure.

Matt Thompson provided a really good answer to the “what’s acceptable oversubscription ratio in a ToR switch” when he wrote “I’m expecting a ‘how long is a piece of string’ answer” (note: do watch the BBC video answering that one).

There’s the 3:1 rule-of-thumb recipe, with a more realistic answer being “it depends”. Now let’s see if we can go beyond that without a deep dive into scholastic waters.

I had an interesting conversation with Doug Hanks (@douglashanksjr) about the need for intra-spine links in leaf-and-spine fabric designs. You clearly don’t need links between spine switches when every leaf node (switch or router/firewall/load balancer) is connected to all spine switches ... but what happens when one of the leaf-to-spine links fails? Will other leaf switches know that they have to avoid the spine switch with the failed link?

Cisco launched two new data center switches on Monday: Nexus 6001, a 1RU ToR switch with the exact same port configuration as any other ToR switch on the market (48 x 10GE, 4 x 40GE usable as 16 x 10GE) and Nexus 6004, a monster spine switch with 96 40GE ports (it has the same bandwidth as Arista’s 7508 in a 4RU form factor and three times as many 40GE ports as Dell Force10 Z9000).

Apart from slightly higher port density, Nexus 6001 looks almost like Nexus 5548 (which has 48 10GE ports) or Nexus 3064X. So where’s the beef?