Pete Lumbis started his Cumulus Linux 4.0 update with an overview of differences between Cumulus Linux on hardware switches and Cumulus VX, and continued with an in-depth list of ASIC families supported by Cumulus Linux.
You can watch his presentation, as well as the more in-depth overview of Cumulus Linux concepts by Dinesh Dutt, in the recently-updated What Is Cumulus Linux All About video.
A few weeks ago I described the basics of AWS networking, now it’s time to describe how different Azure is.
As always, it would be best to watch my Azure Networking webinar to get the details. This blog post is the abridged CliffsNotes version of the webinar (and here’s the reason I won’t write a similar blog post for other public clouds ;).
A while ago we discussed a software-focused view of Network Interface Cards (NICs) with Luke Gorrie, and a hardware-focused view of them with Or Gerlitz (Mellanox), Andy Gospodarek (Broadcom) and Jiri Pirko (Mellanox).
Why would anyone want to implement features in hardware and not in software, and what would be the best hardware implementation? We discussed these dilemmas with Silvano Gai in Episode 110 of Software Gone Wild podcast.
There was an obvious invisible elephant in the virtual Cloud Field Day 7 (CFD7v) event I attended in late April 2020. Most everyone was talking about AWS, how their stuff runs on AWS, how it integrates with AWS, or how it will help others leapfrog AWS (yeah, sure…).
Although you REALLY SHOULD watch my AWS Networking webinar (or something equivalent) to understand what problems vendors like VMWare or Pensando are facing or solving, I’m pretty sure a lot of people think they can get away with CliffsNotes version of it, so here they are ;)
In my quest to understand how much buffer space we really need in high-speed switches I encountered an interesting phenomenon: we no longer have the gut feeling of what makes sense, sometimes going as far as assuming that 16 MB (or 32MB) of buffer space per 10GE/25GE data center ToR switch is another $vendor shenanigan focused on cutting cost. Time for another set of Fermi estimates.
Let’s take a recent data center switch using Trident II+ chipset and having 16 MB of buffer space (source: awesome packet buffers page by Jim Warner). Most of switches using this chipset have 48 10GE ports and 4-6 uplinks (40GE or 100GE).
You haven’t mentioned Intel's Omni-Path at all. Should I be surprised?
While Omni-Path looks like a cool technology (at least at the whitepaper level), nobody ever mentioned it (or Intel) in any data center switching discussion I was involved in.
A while ago we did a podcast with Luke Gorrie in which he explained why he’d love to have simple, dumb, and easy-to-work-with Ethernet NICs. What about the other side of the coin – smart NICs with their own CPU, RAM and operating system? Do they make sense, when and why would you use them, and how would you integrate them with Linux kernel?
In Episode 98 we focused on another interesting application developed by Max Rottenkolber: high-speed VPN gateway using IPsec on top of Snabb Switch (details). Enjoy!
Craig Weinhold sent me his thoughts on using Cisco ACI to implement cross-data-center L4-7 services. While we both believe this is not the way to do things (because you should start with proper application architecture), you might find his insights useful if you have to deal with legacy environments that believe in Santa Claus and solving application problems with networking infrastructure.
An “easy button” for multi-DC is like the quest for the holy grail. I explain to my clients that the answer is right in front of them – local IP addressing, L3 routing, and DNS. But they refuse to accept that, draw their swords, and engage in a fruitless war against common sense. Asymmetry, stateful inspection, ingress routing, split-brain, quorums, host mobility, cache coherency, non-RFC complaint ARP, etc.
In 2014, we did a series of podcasts on Snabb Switch (Snabb Switch and OpenStack, Deep Dive), a software-only switch delivering 10-20 Gbps of forwarded bandwidth per x86 core. In the meantime, Snabb community slowly expanded, optimized the switching code, built a number of solutions on top of the packet forwarding core, and even forked a just-in-time Lua compiler to get better performance.
One of my readers found this Culumus Networks article that explains why you can’t have more than a few hundred VXLAN-based VLAN segments on every port of 48-port Trident-2 data center switch.
Expect to see similar limitations in most other chipsets. There’s a huge gap between millions of segments enabled by 24-bit VXLAN Network Identifier and reality of switching silicon. Most switching hardware is also limited to 4K VLANs.
We started with simple questions like “what is an interface” and “how do they get such weird names in some Linux distributions” which quickly turned into a complex discussion about kernel objects and udev, and details of implementing logical interfaces that are associated with ASIC front-panel physical ports.
Regardless of how much I write about (the ridiculousness of using) stretched VLANs, I keep getting questions along the same lines. This time it’s:
What type of applications require L2 Extension and L3 extension?
I don’t think I’ve seen anyone use L3 extension (after all, isn’t that what Internet is all about), so let’s focus on the first one.
Stretched VLANs (or L2 extensions) are used to solve a number of unrelated problems, because once a vendor sold you a hammer everything starts looking like a nail, and once you get used to replacing everything with nails, you want to use them in all possible environments, including public and hybrid clouds.
After describing the basics of internal data center switch architectures, JR Rivers focused on the crux of the problem the vendors copiously exploit to create a confusopoly: is it better to use big- or small-buffer switches?
You’ll need at least free ipSpace.net subscription to watch the video.
I'm involved in a Nexus 9500 (NX-OS) migration project, and one bug recently caused vPC-connected Catalyst switches to err-disable (STP channel-misconfig) their port-channel members (CSCvg05807), effectively shutting down the network for our campus during what was supposed to be a "non-disruptive" ISSU upgrade.
Weird, right? Wait, there’s more…