… updated on Tuesday, May 17, 2022 14:31 UTC
Networking Hardware/Software Disaggregation in 2022
I started preparing the materials for the SDN – 10 years later webinar, and plan to publish a series of blog posts documenting what I found on various aspects of what could be considered SDN1. I’m pretty sure I missed quite a few things; your comments are most welcome.
Let’s start with an easy one: software/hardware disaggregation in network devices.
Open-Source Network Operating Systems
I found several widely-used open-source2 network operating systems:
- Cumulus Linux
- Dell OS10 (seems to be a successor of Force10 FTOS)
- VyOS
- SONiC
It seems that most of these systems run on a variety of switches from whitebox- and traditional data center switching vendors.
Then there are things likes FBOSS and DANOS3. Is anyone (apart from their creator) using them? I have no idea.
Linux Device Drivers
You need at least two software components to glue a network operating system to the hardware4:
- ASIC driver (sometimes called abstraction layer)
- Platform drivers (something to control the fans, front-panel LEDs…)
There are two competing approaches to ASIC device drivers:
- Switch Abstraction Interface (SAI) – standardized ASIC programming API embraced by Open Compute Project (OCP)
- switchdev – a Linux kernel API that offloads data plane processing from Linux kernel to an ASIC.
Open Network Linux (ONL) includes a large number of platform drivers.
Finally, there’s Stratum from Open Networking Foundation. If I got it right, ONF dropped OpenFlow and focused on P4, which works best on ASICs with flexible forwarding pipeline like the Barefoot/Intel Tofino ASIC5. No wonder the majority of the Technical Steering Team members work for Intel.
Open-Source Operating Systems on Hardware from Traditional Vendors
Most of the traditional data center switching vendors had to support SONiC or offer SAI interface on their hardware, or they wouldn’t be able to sell their boxes to hyperscalers (or at least Microsoft):
- Arista supports SONiC on a wide variety of switches using Tomahawk6, Trident7, Jericho8, or Tofino9 chipsets.
- Dell supports SONiC on switches using Tomahawk and Trident chipsets.
- Juniper supports SONiC on two spine switches using Tomahawk chipset.
- Cisco supports SAI on some Nexus 9200 and Nexus 9300 switches, which means you can run SONiC on them. They also support SONiC on Cisco 8000 routers.
Proprietary Network Operating Systems on Whitebox Hardware
The previous section should have made it abundantly clear that traditional networking vendors don’t mind selling disaggregated hardware (without their software) to large customers. Are they also willing to sell their software to run on third-party hardware? You bet:
- You could run Cisco IOS-XR on third-party hardware
- Juniper is talking about disaggregated Junos, but all I could find was a way to run Junos VM on their NFX150 CPE platform, and a datasheet claiming you can run Junos on a single Accton Edgecore switch.
- Supposedly you could run Arista cEOS (EOS in a container) on third-party whitebox switches. Based on my recent cEOS experience I have to wonder how much functionality you’d get beyond the basic L2+L3 forwarding. The only other reasonable hit I got for “Arista EOS whitebox” was a pointer to my 2015 April 1st blog post.
Then there’s a plethora of niche vendors offering their network operating systems on whitebox hardware, including Arrcus (ArcOS), DriveNets (DNOS), IP Infusion (OcNOS), NoviFlow (NoviWare)10, Pluribus, and RtBrick.
Proprietary Control Plane in a VM or Container
Imagine you’ve used gear from vendor X for ages, and want to deploy new control-plane functionality (example: BGP route reflector for EVPN). Wouldn’t it be better to buy the control plane functionality you need in VM or container format than to be forced to buy a router or a switch even though you need a single port on the device?11
Networking vendors started offering VM versions of their operating systems years ago. You can get (in alphabetical order):
- Arista vEOS
- Cisco IOS XE, IOS XR, or Nexus OS (9000v)
- Cumulus VX
- Dell OS10
- Juniper vSRX, vMX, or vQFX
- Mikrotik RouterOS
- Nokia SR OS and SR Linux
For more details, see also netlab supported platforms.
Some vendors went a step further and offered their control plane in a container. Arista cEOS and Juniper cRPD are the best-known examples.
Revision History
2022-05-17
- Added a pointer to DANOS, DriveNets and a podcast mentioning switchdev
- Added the Proprietary Control Plane in a VM or Container section
Have I missed something?
Your comments (preferably including links to documentation) would be most welcome. In case you want to send me a private message, you already have my email address if you have an ipSpace.net subscription, or if you’re subscribed to my SDN/automation mailing list, and there’s the Contact Us form for everyone else.
-
There’s no need to argue what SDN means, we all know it means Still Don’t Know. ↩︎
-
Using whatever definition of open. ASIC device drivers are often shipping as a binary blob. ↩︎
-
Seems to be a failed AT&T’s attempt to get other people to write software for free… considering the last news were published in 2019. No wonder when the “about the project” link downloads five pages of PDF-ed legalese. ↩︎
-
Assuming we’re dealing with a platform that uses an ASIC for hardware-based packet forwarding ↩︎
-
Even though I found a presentation claiming you can use P4 to program switches with fixed forwarding pipeline. On a totally unrelated topic, you can also run Unix on PDP-11 emulated in JavaScript. ↩︎
-
High-speed Broadcom ASIC used on spine switches ↩︎
-
Feature-rich Broadcom ASIC used on leaf switches ↩︎
-
Broadcom ASIC with large buffers and forwarding tables ↩︎
-
Barefoot (now Intel) ASIC with programmable forwarding pipeline and P4 support. ↩︎
-
NoviWare seems to be an OpenFlow agent, not a full-blow network operating system. ↩︎
-
Or two for redundancy ;) ↩︎
You could add Drivenets under the proprietary NOS on top of white boxes. AT&T seems to have deployed this in their next-gen core or something.
https://drivenets.com/news-and-events/press-release/att-deploys-drivenets-network-cloud-in-their-next-gen-core/
Thank you, will add.
Although, is this the same AT&T that tried really hard to persuade everyone else to do their job for them by open-sourcing DANOS? 🤣
Thanks a million, really helped me get a slide or two more focused (obviously with full attribution).
P4 is not just for programming. It is intended to be used as generic pipeline architecture description language. The various pipeline architectures started in ONOS, but then they were tired of it and wanted to move downwards closer to the hardware, so the SDN controller would be more unified. Google had the power to enforce this, so Stratum was born. ONOS has a new variant written in GoLang called MicroONOS. This is a streamlined version that supports only P4 as a southbound API. So the SDN contoller would not have a zoo of drivers and a zoo of pipeline modules. The configuration complexity is now moved away from the SDN operator to the switch designer. However, the SDN controller code will be much more complex, since it needs to interpret P4. But the SDN operator has less work to do. If an ASIC is not programmable, you could still write a generic network operation system using P4. This is the theory, but for simpler hardware OpenFlow is good enough and there are mature implementations. The P4 SDN development is rather slow, since there is not a big business drive for it. Some companies changed from open systems to closed systems. For example, HPE. They have recognized that many companies do not care, they just need a simple to operate solution. Interoperability with different vendors is just for nerds or telcos. Not for the average enterprise. They are happy with a black box, single vendor solution if it works enough well...
Thanks a million for the feedback -- so it's yet again the question of "where do you want to hide the complexity" ;)
Hi, Ivan! Very nice article, thank you! May I ask about the use-cases you mentioned? At the end of the article you said:
Feature-rich Broadcom ASIC used on leaf switches,
High-speed Broadcom ASIC used on spine switches
The other article from Juniper I found on the web (https://www.juniper.net/content/dam/www/assets/white-papers/us/en/routers/deploying-data-center-switching-solutions.pdf) was saying quite opposite:
Low On-Chip Memory ASIC: High speed/low buffer leaf
Large External Memory ASIC: High speed/high buffer leaf and spine
Here (https://elegantnetwork.github.io/posts/A-Summary-of-Network-ASICs/) it says that "Historically these chips (merchant silicon) have been used for ToR switches, but some networks especially hyperscalers are using them throughout the network, usually in the form of small fixed form factor (pizza boxes.)"
I guess there is a naming confusing (e.g. do we call ToR as Leaf or no). May I ask you to clarify what is the best use-case of each type of ASIC?
Memory and feature richness are orthogonal issues, you can have low memory and feature richness. Typically in a lean spine fabric design you have features on the edge (leaf). Only leafs terminating external and security services typically would need high buffer so you can use the right ASIC for the role.
Great article Ivan! With the cloud evolution the last 10 years the entire definition of networking has evolved. Networking is a service that can run anywhere depending on customer requirement. The ability to run a mature BGP or even L7 Security stack on a containerized platform in a on premise server or cloud workload shows how flexible infrastructure has become. Here's Juniper's cRPD documentation https://www.juniper.net/documentation/us/en/software/crpd/crpd-deployment/topics/concept/understanding-crpd.html
Thank you - totally missed the "control plane in VM/container" aspect. Added.
Yes, cRPD runs since a bit on top of wet string up to whitebox HW and probably even funkier stuff ;-) Ivan, I recommend to grab someone like Emanuel or one of our key SEs dealing with cRPD in different envs and do a little podcast just about this stuff. AFAIS the future points to something like cRPD as most succesful approach given how small/cheap/fast-boot the container is (and yes, southbound API evreyone is so excited about is largely circumstantial, cRPD simply chopped off the southbound [very rich] JNPR ASIC API and replaced it with bunch other things with cost of adding other southbound APIs being fairly cheap though mostly loosing good amount of functionality since such APIs [and off the shelf ASICs or even Linux fwd path] are much less feature rich than J's silicon. As theyu say, what you pay for but generally it's just one layer API squeezing a whale into a needle often ;-) ... why I say future points that way is that the real hard stuff in integraion of all that stuff iis really in northbound (like netconf or yang)/telemetry/JET API and so on. It's sexy t o talk about building a "better BGP" but once you have it you have nothing yet in a large network that needs automating/operating/unified, large scale solid northbounds. And once you really start to look at it servers/hosts start to slide into it for a couple reasons (as in RotH) the discussion of scale/running stuff on host securely and so on points hard towards something like cRPD again. my 2c.
Thanks a million (as always) for the feedback -- it's really nice to see I got it mostly right (although I will need another blog post to explain it).
In SONIC the SAIs are proprietary but I guess you can download them for free. In Cumulus, switchd is proprietary and you can't get it for free; I'm not sure if such a "closed core" model is what people expect from open source.
You're absolutely right, and that's why I wrote "using whatever definition of open" 🥴
NVIDIA Linux Switch is an interesting approach. All the hardware drivers are included in the stock Linux kernel so you can install any recent Linux distribution and use Linux applications from the distro for routing, monitoring, and configuration management (e.g. FRR, Host sFlow, Ansible, etc.)
I love switchdev but 99% of network engineers/operators cannot handle it because there's no vendor support/education/ecosystem. I guess we can view it as a discount for open-minded people. :-)
Is that still true? A lot of data center network edge functionality has moved to the Linux hosts to support virtualization / containers so there are many operators with Linux / network crossover skills. Having a single NOS end to end greatly simplifies end to end troubleshooting.
Ivan - with all the traction which SONiC is getting, I would love to add some colors on the multi-vendor SONiC deployment/Support and tier-2 adoption for it. Thanks, Vishal
There's PicOS from Pica8. They've presented at a couple Networking Field Days.
We should also add reference to DENTos - targeting edge-usecases (Retail networking, campus, etc). Major driving force behind DENTos is Amazon.