What Happened to FabricPath and Its Friends?
Continuing the what happened to old technologies saga, here’s another question by Enrique Vallejo:
Are FabricPath, TRILL or SPB still alive, or has everyone moved to VXLAN? Are they worth studying?
TL&DR: Barely. Yes. No.
Layer-2 Fabric craziness exploded in 2010 with vendors playing the usual misinformation games that eventually resulted in totally fragmented market full of partial- or proprietary solutions. At one point in time, some HP data center switches supported only TRILL, and other data center switches from the same company supported only SPB.
Now for individual technologies:
- I wrote the last major blog post about TRILL in 2012. As Gartner would say, TRILL died before it reached the through of disillusionment1.
- Juniper QFabric was the first to die2. Scaling a centralized control and management plane to 128 switches is a hard task, and even though Juniper might have been best-positioned to make it work (and had decent architecture), in the end they still failed.
- Juniper Virtual Chassis was a lame attempt to make stackable switches work in data center environment, and even Virtual Chassis Fabric couldn’t save the day. Running a single control plane for the whole fabric on an underpowered switch CPU is obviously not a good idea (see also: OpenFlow).
- When Nexus 9000 switches were launched without FabricPath support in 2013, the writing on the wall was very obvious. I wouldn’t be surprised to see FabricPath deployments in the wild, but anyone considering it for a Greenfield deployment should look at the calendar.
- Brocade was the last data center switching vendor worth mentioning that abruptly joined the EVPN/VXLAN lemming run in 2016, dropping VCS Fabric like a steaming pile of ****.
- HP3 was overpromising and underdelivering until they managed to become irrelevant. The were telling us they planned to support TRILL, SPB, and FCoE at Networking Field Day 1 in 2010, and couldn’t deliver a single one of those protocols by the time of Nexus 9000/ACI launch. They did manage to deliver in the end, but it was already too late. According to Philipp, HPE is still selling H3C-developed switches with Comware version 7 that supports VXLAN with EVPN. They also push Aruba switches these days, and while (according to data sheets) they support VXLAN/EVPN, there’s zero mention of TRILL or SPB.
How about standard-based layer-2 fabric implementations? I’m not aware of anyone (apart from HPE) ever implementing TRILL4. Avaya had interesting SPBM implementation (more details), and they’re still selling SPBM switches under the Extreme brand. I’ve heard someone mentioning a SP-focused vendor (Alcatel Lucent?) using SPB in Carrier Ethernet implementations a while ago, but I never looked at the details.
I probably missed a few things, in which case a comment from someone with more details (preferably with links to product documentation) would be most welcome.
Revision History
- 2022-05-03
- Added more information on HPE data center switches
- 2022-05-04
- Added a footnote pointing to a TRILL implementation.
-
Gartner made a similar claim about SDN in its 2021 Networking Hype Cycle document. I’m not sure product marketers working for networking vendors got the memo. ↩︎
-
More precisely, a migration document from 2019 says “Juniper Networks continues to support QFabric Systems, but many of the hardware devices in QFabric Systems have reached end of life (EOL) status.” ↩︎
-
An entity now known as HPE because someone realized pouring huge profits from toner packages into developing products that consistently stay in the “others” category in industry surveys makes no sense. I couldn’t figure out what happened with H3C, the company developing the data center switches HP/HPE was selling under their own brand until the Aruba acquisition. ↩︎
-
However, there’s always someone saying “I could use this technology to solve X”. According to Kevin Myers, at least one company uses TRILL in their fixed wireless mesh products. ↩︎
Hi Ivan,
as for footnote #3: I don't know what happend to the H3C company, but the products are still alive under HPE. Named "FlexFabric" (Datacenter) and "FlexNetworks" (Campus). The still run the ComwareOS (Version 7 now) and support EVPN/VXLAN. A hardware refresh happend last autumn. I know a few customer who use them in the campus or 'datacenter' (10ish racks + VLANs) and the whole reason they use them is because "We buy them every time". Aruba launched a new SwitchOS (Aruba AOS-CX) a few years ago and they aim to built one OS for Access to Core to Datacenter. Works quite well for standard use cases including EVPN/VXLAN. BTW: Both Comware 7 and Aruba AOS-CX are available for virtualization, the last one free of charge.
Thank you! Updated the blog post.
It would be good to understand why TRILL/SPB failed so we can avoid making the same mistake again. It seems that network vendors were blindsided by the much faster development velocity of the hypervisor/OS/cloud vendors.
Hi Ivan
The extreme SPBM is alive and kicking. You should ask Roger Lapuh for update…Extreme evolve the solution with ZTP Fabric ,ISIS Multi area …factors from CPE size to chassis. Yes they do something different and that’s IMO refreshing!!
Yeah, haven't seen Roger for ages :( Time to fix that, I guess...
The old Avaya material was excellent !!but need some refresh…
I specially liked the software gone wild podcast very good discussion …
The Alcatel-Lucent Enterprise (ALE) OmniSwitch series and Extreme Networks VSP/ERS series (formerly Avaya and before that Nortel) are still supporting SPB with active production deployments. Extreme did not kill it after the Avaya acquisition and is still pushing it as Extreme FabricConnect. ALE is more quiet, but isn't showing any signs of deprecating it either. Both vendors support additional L3VPN capabilities on top of L2 virtualization regular 802.1aq/SPB provides. I wouldn't count on full interoperability between their implementations though.
Nokia inherited an SPB implementation from the Alcatel-Lucent Service Provider line (SR series). I don't know if there are/were any significant production deployments. HPE/H3C had an implementation on some ComWare gear, but it's considered a legacy feature.
Standardization efforts in the IETF seem to have halted completely (e.g. https://datatracker.ietf.org/doc/draft-unbehagen-spb-ip-ipvpn/ and https://datatracker.ietf.org/doc/draft-unbehagen-lldp-spb/). I don't if that's due to IETF politics or the vendors simply losing interest. On the IEEE side, at least some efforts seem to be going on https://1.ieee802.org/tsn/802-1qcj/ but who knows, as IEEE is mostly a walled garden.
Although SPB is still a Layer 2 technology, there is a certain charm about its elegance and simplicity: A single IS-IS control plane for STP/Unicast Underlay Routing/Multicast Underlay Routing/Unicast Overlay Routing/Multicast Overlay Routing. VXLAN EVPN is much much more complicated in this regard.
At least in campus networks SPB is still worth a consideration, if you're willing to accept niche vendors. VXLAN EVPN hasn't really reached budget-friendly access switches yet, there are budget-friendly SPBM switches available though.
I definitely agree that SPBM with extensions nicely solves some challenges in less complex ways than the more traditional stacks... assuming you don't grow too large, and don't need routing integration with the outside world (SPBM to PIM gateway must be great fun).
As for "budget-friendly switches", PBB is a much simpler data plane (fewer lookups needed at tunnel egress), and it got implemented in older/simpler (and now cheaper) chipsets.
When it comes to H3C, skimming over documentation reveals that just VxLAN is mentioned, not TRILL.
Removed Ethernet switching from the fabric -- QF used Broadcom's HIGIG frame format between fabric nodes to enable tagged forwarding in the fabric. The ingress Node (Leaf) performed a lookup on the destination MAC and marked the HIGIG header field with a "tag" that represented the destination switch. The Interconnect (spine) layer performed forwarding on the data in the HIGIG field, not on the original Ethernet frame fields.
All of the forwarding intelligence was contained at the Node (leaf) layer. All forwarding decisions, both L2 and L3, were performed at the ingress Node (leaf). The Interconnects (spines) were simply lean forwarders.
All MAC and ARP learning happened in the QF control plane. The system employed BGP and the MAC VPN draft (https://datatracker.ietf.org/doc/html/draft-raggarwa-mac-vpn-01) for MAC and ARP learning.
It was surely a science project, and something that still makes customers and SE's cringe on occasion. But I don't think we'd be where we are without some of the capabilities that QF demonstrated.
While I agree with everything you wrote (see also my earlier posts on QFabric), none of those ideas were new:
I do agree though that BGP had not been used for MAC addresses before, and that draft is indeed a precursor of EVPN, but that's not what QFabric was all about. Juniper wanted to build a single-management-entity fabric and they failed miserably.
My last fabricpath deployment is back to 2016, and recently years there're a lot of customers are migrate it to VXLAN/EVPN. You'll be surprised that how wildly people have deployed this technology. They really use it to extend the L2 fabric everywhere across DCs. Now it'll be a nightmare for us to migrate it to new fabric infrastructures.