Juniper has finally released the technical documentation for the QFabric virtual switch and its components (QF/Node, QF/Interconnect and QF/Director). As expected, my speculations weren’t too far off – if anything, Juniper didn’t go far enough along those lines, but we’ll get there later.
The generic hardware architecture of the QFabric switching complex has been well known for quite a while (listening to the Juniper QFabric Packet Pushers Podcast is highly recommended) – here’s a brief summary:
Redundant connections between QFabric elements and control plane stackable switches are not shown in the network diagram. Each QF/Node has two connections to the virtual chassis switches, QF/Interconnect has four (two per control board), QF/Director has six (three per network module).
QF/Directors are x86-based devices that act like the brains for the QFabric, providing fabric services (management, configuration, control, device discovery, DNS, DHCP, NFS) and routing engines for more complex node clusters (network node groups).
Each QFabric should have at least two QF/Directors with disks; you can add diskless QF/Directors (no SKU yet) if you need more processing power (not likely in the current software release).
QF/Interconnects (QFX3008) are very-high-speed totally proprietary switches that forward frames exchanged between QF/Nodes. Each QFX3008 provides up to 10Tbps of non-blocking bandwidth (where non-blocking is defined as “any input port can send a packet to any non-busy output port”) and uses three-stage Clos network to get the non-blocking behavior. With up to four QF/Interconnects per QFabric, the total QFabric switching bandwidth is 40Tbps. Impressive. Try to calculate how many 64 kbps voice calls can fit into that ;)
QF/Nodes are the well-known QFX3500 L2/L3 switches. They support 10GE, 2/4/8Gb FC and up to 4 40GE uplinks to the QF/Interconnect. With 48 10GE ports, you get 1:3 oversubscription if you use all four uplinks or 1:6 oversubscription if you decide to use only two uplinks (using only one uplink probably doesn’t make much sense).
QFabric uses out-of-band Control plane LAN implemented with two stacks of EX4200 switches. Each QFabric component has redundant connections to the control-plane LAN (QF/Node has one connection to each Virtual Chassis, QF/Interconnect has two, QF/Director three). All control-plane traffic is exchanged on the control-plane LAN (already getting ATM/SDH/MPLS-TP flashbacks?), nicely isolating it from the user traffic. The QF/Director has separate management and control plane ports, making the control plane LAN totally isolated.
1-tier? Really? Looking at the QFabric architecture, one has to wonder why Juniper claims it’s a 1-tier architecture. Honestly, it’s as much 1-tier as every MPLS/VPN network I’ve ever seen. However, like with MPLS/VPN, there’s a trick – QFabric uses single-lookup forwarding.
The ingress QF/Node performs full L2/L3 lookup (including ACL checks) and decides how to forward the packet to the egress QF/Node. The QF/Interconnect uses the proprietary frame forwarding information to get the user data to the egress QF/Node. The frame forwarding information likely includes enough details to allow the egress QF/Node to forward the frame to the output port.
The expensive part of the user frame/packet lookup is thus performed only once (whereas you’d get three full lookups in a traditional data center design using similar hardware architecture). Net result: 5 microsecond forwarding latency across the fabric. Not bad, considering that the QF/Interconnect itself has three hops.
Once you get over the totally proprietary nature of QFabric, the initial commitment you have to make (according to this post, the minimum you’d pay for a single QF/Interconnect with two linecards and two QF/Directors would be around $450.000 ... without optics or a single QF/Node) and the amount of lock-in you’d be exposed to (with all other vendors, you can slowly phase in or out of their fabrics; with QFabric it’s all-or-nothing), QFabric is indeed a masterpiece of engineering.
Due to all the above-mentioned facts, I would expect to see it deployed primarily in very large Greenfield environments; huge Hadoop/MapReduce clusters immediately come to mind.
The Juniper QFabric Packet Pushers Podcast is probably still the best independent source of information on QFabric hardware architecture and its data plane.
I’ll talk about data center fabric architectures and networking requirements for cloud computing at the upcoming EuroNOG conference.
Fabric-like architectures from various vendors are the main focus of the Data Center Fabric Architectures webinar.
You’ll find in-depth discussions of various data center and network virtualization technologies in Data Center 3.0 for Networking Engineers webinar (recording), which is also part of the Data Center Trilogy.
Both webinars (and numerous others) are included in the yearly subscription.