Category: switching

A while ago we discussed a software-focused view of Network Interface Cards (NICs) with Luke Gorrie, and a hardware-focused view of them with Or Gerlitz (Mellanox), Andy Gospodarek (Broadcom) and Jiri Pirko (Mellanox).

Why would anyone want to implement features in hardware and not in software, and what would be the best hardware implementation? We discussed these dilemmas with Silvano Gai in Episode 110 of Software Gone Wild podcast.

There was an obvious invisible elephant in the virtual Cloud Field Day 7 (CFD7v) event I attended in late April 2020. Most everyone was talking about AWS, how their stuff runs on AWS, how it integrates with AWS, or how it will help others leapfrog AWS (yeah, sure…).

Although you REALLY SHOULD watch my AWS Networking webinar (or something equivalent) to understand what problems vendors like VMWare or Pensando are facing or solving, I’m pretty sure a lot of people think they can get away with CliffsNotes version of it, so here they are ;)

In my quest to understand how much buffer space we really need in high-speed switches I encountered an interesting phenomenon: we no longer have the gut feeling of what makes sense, sometimes going as far as assuming that 16 MB (or 32MB) of buffer space per 10GE/25GE data center ToR switch is another $vendor shenanigan focused on cutting cost. Time for another set of Fermi estimates.

Let’s take a recent data center switch using Trident II+ chipset and having 16 MB of buffer space (source: awesome packet buffers page by Jim Warner). Most of switches using this chipset have 48 10GE ports and 4-6 uplinks (40GE or 100GE).

One of my subscribers sent me this question after watching the latest batch of Data Center Fabrics videos:

You haven’t mentioned Intel's Omni-Path at all. Should I be surprised?

While Omni-Path looks like a cool technology (at least at the whitepaper level), nobody ever mentioned it (or Intel) in any data center switching discussion I was involved in.

A while ago we did a podcast with Luke Gorrie in which he explained why he’d love to have simple, dumb, and easy-to-work-with Ethernet NICs. What about the other side of the coin – smart NICs with their own CPU, RAM and operating system? Do they make sense, when and why would you use them, and how would you integrate them with Linux kernel?

We discussed these challenges with Or Gerlitz (Mellanox), Andy Gospodarek (Broadcom) and Jiri Pirko (Mellanox) in Episode 99 of Software Gone Wild.

In previous Software Gone Wild episodes we covered Snabb Switch and numerous applications running on it, from L2VPN to 4over6 gateway and integration with Juniper vMX code.

In Episode 98 we focused on another interesting application developed by Max Rottenkolber: high-speed VPN gateway using IPsec on top of Snabb Switch (details). Enjoy!

Craig Weinhold sent me his thoughts on using Cisco ACI to implement cross-data-center L4-7 services. While we both believe this is not the way to do things (because you should start with proper application architecture), you might find his insights useful if you have to deal with legacy environments that believe in Santa Claus and solving application problems with networking infrastructure.

An “easy button” for multi-DC is like the quest for the holy grail. I explain to my clients that the answer is right in front of them – local IP addressing, L3 routing, and DNS. But they refuse to accept that, draw their swords, and engage in a fruitless war against common sense. Asymmetry, stateful inspection, ingress routing, split-brain, quorums, host mobility, cache coherency, non-RFC complaint ARP, etc.  

In 2014, we did a series of podcasts on Snabb Switch (Snabb Switch and OpenStack, Deep Dive), a software-only switch delivering 10-20 Gbps of forwarded bandwidth per x86 core. In the meantime, Snabb community slowly expanded, optimized the switching code, built a number of solutions on top of the packet forwarding core, and even forked a just-in-time Lua compiler to get better performance.

To find out the details, listen to Episode 91 of Software Gone Wild in which Luke Gorrie explained how far the Snabb project has progressed in the last four years.

One of my readers found a Culumus Networks article that explains why you can’t have more than a few hundred VXLAN-based VLAN segments on every port of 48-port Trident-2 data center switch. That article has unfortunately disappeared in the meantime, and even the Wayback Machine doesn’t have a copy.

Continuing the Linux networking discussion we had in Episode 86, we focused on Linux interfaces in Episode 87 of Software Gone Wild with Roopa Prabhu and David Ahern.

We started with simple questions like “what is an interface” and “how do they get such weird names in some Linux distributions” which quickly turned into a complex discussion about kernel objects and udev, and details of implementing logical interfaces that are associated with ASIC front-panel physical ports.

Regardless of how much I write about (the ridiculousness of using) stretched VLANs, I keep getting questions along the same lines. This time it’s:

What type of applications require L2 Extension and L3 extension?

I don’t think I’ve seen anyone use L3 extension (after all, isn’t that what Internet is all about), so let’s focus on the first one.

Stretched VLANs (or L2 extensions) are used to solve a number of unrelated problems, because once a vendor sold you a hammer everything starts looking like a nail, and once you get used to replacing everything with nails, you want to use them in all possible environments, including public and hybrid clouds.

After describing the basics of internal data center switch architectures, JR Rivers focused on the crux of the problem the vendors copiously exploit to create a confusopoly: is it better to use big- or small-buffer switches?

Looks like I’m becoming the gateway-of-last-resort for people encountering totally weird Nexus OS bugs. Here’s another beauty…

I'm involved in a Nexus 9500 (NX-OS) migration project, and one bug recently caused vPC-connected Catalyst switches to err-disable (STP channel-misconfig) their port-channel members (CSCvg05807), effectively shutting down the network for our campus during what was supposed to be a "non-disruptive" ISSU upgrade.

Weird, right? Wait, there’s more…

I haven’t done an update on what Avaya was doing in the data center space for years, so I asked my good friend Roger Lapuh to do a short presentation on:

• Avaya’s data center switches and their Shortest Path Bridging (SPB) fabric;
• SPB fabric features;
• Interesting use cases enabled by SPB fabric.

The videos are now available to everyone with a valid ipSpace.net account – the easiest way to get it is a trial subscription.

A while ago (in the time of big-versus-small buffers brouhaha), I asked JR Rivers to do a short presentation focusing on buffering requirements of data center switches. He started by describing typical buffer architectures you might find in data center switches.