A tweet from J Michel Metz has alerted me to a “Why TRILL won't work for data center network architecture” article by Anjan Venkatramani, Juniper’s VP of Product Management. Most of the long article could be condensed in two short sentences my readers are very familiar about: Bridging does not scale and TRILL does not solve the traffic trombone issues (hidden implication: QFabric will solve all your problems)... but the author couldn’t resist throwing “FCoE over TRILL” bone into the mix.

I thought that bone has been scrapped clean in the last few months, but it seems some people still think it’s worth chewing. Let’s try to make it really simple. There are two fundamental ways of implementing multi-hop FCoE:

• Dense-mode FCoE, where every LAN switch is also Fiber Channel Forwarder (FCF; equivalent to an IP router). FCoE frames are routed hop-by-hop and FCoE routing does not interact with STP or TRILL (because FCoE is not bridged). It’s also easy to implement SAN A/SAN B separation (two independent non-overlapping paths from each server to each storage device) as you control hop-by-hop flow of the FC traffic.
• Sparse-mode FCoE, where LAN switches bridge FCoE frames like any L2 traffic and optionally provide FIP snooping. Because FCoE is bridged in this design, sparse-mode FCoE needs stable L2 transport which TRILL can provide more reliably than existing STP-based networks. Since you can’t control hop-by-hop FCoE traffic flow (it’s bridged according to whatever TRILL or STP thinks is the best path), it’s way harder to implement the SAN separation.

Got it? How hard was it? Do I really need to spell it out every few months?

Note to the equipment buyers: every time a vendor mentions “FCoE over TRILL” ask yourself “are they trying to divert my attention?” Most probably they don’t have full-blown FCF code in their switches.

Obviously I had to check QFX3500 JunOS documentation after reading the afore-mentioned article. As I came to expect from Juniper, the documentation is precise and well-written. QFX3500 supports FC and FCoE, has up to 12 “universal” ports (two groups of 6 ports) that can be configured as 2/4/8 GB FC or 10GE ... but does not have a full-blown L3 FC stack (as I expected). The only marketing intrusion I found in the documentation was the FCoE transit switch, a term which does not appear anywhere in the FC-BB-5 standard – obviously someone thought it sounds better than FIP snooping bridge.

Here’s what QFX3500 can do on the FCoE/FC front:

• It can be a FIP-snooping bridge, forwarding FCoE traffic between its 10GE ports and protecting FCFs and FCoE Enodes with dynamically built ACLs based on FIP requests/responses. Bonus points for the documentation being very clear on the traffic flow: FCoE traffic has to go through a full-blown FCF.
• It can be an FC N_port proxy (using NPIV) for FCoE Enodes. Yet again, as it doesn’t have the FCF code, the traffic between FCoE nodes has to go to a FC-based FCF and back (also clearly documented).
• It supports exactly the same DCB standards as Cisco: PFC, ETS and DCBX, but not QCN. PFC and ETS should be enough unless you build really large bridged networks (hint: don’t).

In most cases, the functionality offered by QFX3500 is more than enough to implement access-layer convergence. If you use its NPIV functionality and send FCoE traffic from the servers straight into FC SAN, you can even maintain true SAN separation with servers attached to two QFX3500 switches. Transporting FCoE across the network toward an FCF (or another QFX3500 acting as NPIV proxy) that’s several hops away should still work, but it’s harder to maintain the strict SAN separation.

Am I allowed to conclude with a note to the vendors’ marketing departments? We might like you more if you tell us how your boxes actually work, what they can do and how we can build great solutions with them instead of constantly harassing us with the arguments why the things you haven’t implemented aren’t rosy, more so if you have a good product that can’t possibly benefit from such tactics ... not that I would ever expect them to listen.

More information

You’ll find more information about FCoE, FIP snooping, FC proxy solutions and Data Center Bridging (DCB) standards in my Data Center 3.0 for Networking Engineers webinar (buy a recording or a yearly subscription).