Building network automation solutions

9 module online course

Start now!

Category: switching

Switch Buffer Sizes and Fermi Estimates

In my quest to understand how much buffer space we really need in high-speed switches I encountered an interesting phenomenon: we no longer have the gut feeling of what makes sense, sometimes going as far as assuming that 16 MB (or 32MB) of buffer space per 10GE/25GE data center ToR switch is another $vendor shenanigan focused on cutting cost. Time for another set of Fermi estimates.

Let’s take a recent data center switch using Trident II+ chipset and having 16 MB of buffer space (source: awesome packet buffers page by Jim Warner). Most of switches using this chipset have 48 10GE ports and 4-6 uplinks (40GE or 100GE).

read more see 8 comments

Smart NICs and Related Linux Kernel Infrastructure

A while ago we did a podcast with Luke Gorrie in which he explained why he’d love to have simple, dumb, and easy-to-work-with Ethernet NICs. What about the other side of the coin – smart NICs with their own CPU, RAM and operating system? Do they make sense, when and why would you use them, and how would you integrate them with Linux kernel?

We discussed these challenges with Or Gerlitz (Mellanox), Andy Gospodarek (Broadcom) and Jiri Pirko (Mellanox) in Episode 99 of Software Gone Wild.

read more see 1 comments

High-Speed IPsec on Snabb Switch on Software Gone Wild

In previous Software Gone Wild episodes we covered Snabb Switch and numerous applications running on it, from L2VPN to 4over6 gateway and integration with Juniper vMX code.

In Episode 98 we focused on another interesting application developed by Max Rottenkolber: high-speed VPN gateway using IPsec on top of Snabb Switch (details). Enjoy!

add comment

Cross-Data-Center L4-7 Services with Cisco ACI

Craig Weinhold sent me his thoughts on using Cisco ACI to implement cross-data-center L4-7 services. While we both believe this is not the way to do things (because you should start with proper application architecture), you might find his insights useful if you have to deal with legacy environments that believe in Santa Claus and solving application problems with networking infrastructure.


An “easy button” for multi-DC is like the quest for the holy grail. I explain to my clients that the answer is right in front of them – local IP addressing, L3 routing, and DNS. But they refuse to accept that, draw their swords, and engage in a fruitless war against common sense. Asymmetry, stateful inspection, ingress routing, split-brain, quorums, host mobility, cache coherency, non-RFC complaint ARP, etc.  

read more see 1 comments

Snabb Switch Update on Software Gone Wild

In 2014, we did a series of podcasts on Snabb Switch (Snabb Switch and OpenStack, Deep Dive), a software-only switch delivering 10-20 Gbps of forwarded bandwidth per x86 core. In the meantime, Snabb community slowly expanded, optimized the switching code, built a number of solutions on top of the packet forwarding core, and even forked a just-in-time Lua compiler to get better performance.

To find out the details, listen to Episode 91 of Software Gone Wild in which Luke Gorrie explained how far the Snabb project has progressed in the last four years.

see 1 comments

VXLAN Limitations of Data Center Switches

One of my readers found this Culumus Networks article that explains why you can’t have more than a few hundred VXLAN-based VLAN segments on every port of 48-port Trident-2 data center switch.

Expect to see similar limitations in most other chipsets. There’s a huge gap between millions of segments enabled by 24-bit VXLAN Network Identifier and reality of switching silicon. Most switching hardware is also limited to 4K VLANs.

read more see 5 comments

Linux Interfaces on Software Gone Wild

Continuing the Linux networking discussion we had in Episode 86, we focused on Linux interfaces in Episode 87 of Software Gone Wild with Roopa Prabhu and David Ahern.

We started with simple questions like “what is an interface” and “how do they get such weird names in some Linux distributions” which quickly turned into a complex discussion about kernel objects and udev, and details of implementing logical interfaces that are associated with ASIC front-panel physical ports.

read more add comment

Revisited: The Need for Stretched VLANs

Regardless of how much I write about (the ridiculousness of using) stretched VLANs, I keep getting questions along the same lines. This time it’s:

What type of applications require L2 Extension and L3 extension?

I don’t think I’ve seen anyone use L3 extension (after all, isn’t that what Internet is all about), so let’s focus on the first one.

Stretched VLANs (or L2 extensions) are used to solve a number of unrelated problems, because once a vendor sold you a hammer everything starts looking like a nail, and once you get used to replacing everything with nails, you want to use them in all possible environments, including public and hybrid clouds.

read more see 14 comments

What Exactly Should My MAC Address Be?

Looks like I’m becoming the gateway-of-last-resort for people encountering totally weird Nexus OS bugs. Here’s another beauty…

I'm involved in a Nexus 9500 (NX-OS) migration project, and one bug recently caused vPC-connected Catalyst switches to err-disable (STP channel-misconfig) their port-channel members (CSCvg05807), effectively shutting down the network for our campus during what was supposed to be a "non-disruptive" ISSU upgrade.

Weird, right? Wait, there’s more…

read more see 4 comments

Video: Avaya [now Extreme] Data Center Solutions

I haven’t done an update on what Avaya was doing in the data center space for years, so I asked my good friend Roger Lapuh to do a short presentation on:

  • Avaya’s data center switches and their Shortest Path Bridging (SPB) fabric;
  • SPB fabric features;
  • Interesting use cases enabled by SPB fabric.

The videos are now available to everyone with a valid ipSpace.net account – the easiest way to get it is a trial subscription.

see 6 comments

Create a VLAN Map from Network Operational Data

It’s always great to see students enrolled in Building Network Automation Solutions online course using ideas from my sample playbooks to implement a wonderful solution that solves a real-life problem.

James McCutcheon did exactly that: he took my LLDP-to-Graph playbook and used it to graph VLANs stretching across multiple switches (and provided a good description of his solution).

add comment

Are VXLAN-Based Large Layer-2 Domains Safer?

One of my readers was wondering about the stability and scalability of large layer-2 domains implemented with VXLAN. He wrote:

If common BUM traffic (e.g. ARP) is being handled/localized by the network (e.g. NSX or ACI), and if we are managing what traffic hosts can send with micro-segmentation style filtering blocking broadcast/multicast, are large layer-2 domains still a recipe for disaster?

There are three major (fundamental) problems with large L2 domains:

read more see 4 comments
Sidebar