FCoE networking elements classification

When I (somewhat jokingly) wrote about the dense- and sparse-mode FCoE, I had no idea someone would try to extend the analogy to all possible FCoE topologies like Tony Bourke did. Anyhow, now that the cat is out of the bag, let’s state the obvious: enumerating all possible FCoE topologies is like trying to list all possible combinations of NAT, IP routing over at least two L2 technologies, and bridging; while it can be done, the best one can reasonably hope for is a list of supported topologies from various vendors.

However, it might make sense to give you a series of questions to ask the vendors offering FCoE gear to help you classify what their devices actually do.

Does it process the FCoE frame based on FC addresses?

No – it’s not really an FCoE device, it’s a fancy Ethernet bridge. The very minimum the bridge must support is ETS (802.1Qaz) and PFC (802.1Qbb) with QCN being optional (unless you want to deploy sparse-mode FCoE with plenty of devices between FCoE nodes).

If the device supports FIP snooping (like Cisco’s Nexus 4000 or Juniper’s QFX3500), it’s marginally more useful and can provide some additional security in sparse-mode FCoE environments (where FCoE nodes are separated by DCB-capable bridges).

Yes – it might actually be an FCoE-capable device. Let’s dig further.

When in doubt check the device documentation (starting with the configuration guides). If it doesn’t tell you how to configure VSANs, NPIV or FSPF, I would remain dubious.

Can it connect to a host CNA through a DCB-capable bridge (not that this would be a good idea)?

No – Too bad. Your connectivity options might be limited.

Yes – Sounds great, but be careful. Check whether you can group the ENodes into VLANs/VSANs based on their MAC addresses (you can do that on Nexus 5000, but not on QFX3500). Without this functionality, you’ll have hard time implementing SAN-A/SAN-B separation when the hosts are not directly connected to the FCoE switches.

In both cases, proceed to the next question.

Does it run FSPF?

No – it’s not really an FCF (FCoE Forwarder), but a gateway running NPIV (in IP lingo: a NAT device with static default route), like Juniper’s QFX3500 or Cisco’s Nexus switches in NPV mode (HP’s Virtual Connect seems to be in this category as well).

It’s also worth asking what the uplink options are. Cisco’s switches (excluding Nexus 6100 for the moment, see comments) support FCoE uplinks, Juniper and HP require FC uplinks.

Yes – congratulations. It’s actually an FC router supporting FCoE. Now let’s get into connectivity details. Oh, and it might be a good idea to check whether NPIV is supported as it just might come handy (it’s not on Brocade’s VDX switches).

Can it connect to another FCF over FCoE (can it create VE_ports)?

No – Too bad. No multihop FCoE today, but it might still be a good option if you want to connect servers with 10GE CNA to existing FC SAN.

Yes – Sounds great. You might have stumbled across a full-blown FCoE FCF. To my knowledge, Cisco’s Nexus family and Brocade’s VDX switches (VCS fabric) are the only two in this category.

Does the packet forwarding follow VE_port-based links and FSPF topology information?

Yes – seems like the FCF you’re looking at follows FC-BB-5 pretty closely.

No – Weird. This is not how FC-BB-5 is written (but it’s how Brocade’s VCS fabric works).

Anyway, this was just a trick question to annoy a few people out there. Let’s move on.

Can you insert DCB-capable bridges between two FCFs (VE_ports running across bridged Ethernet)?

No – No sparse-mode multihop FCoE for you today. Whether that matters is a good topic for another discussion.

Yes – You’ve found a truly versatile FCoE device that supports both sparse-mode and dense-mode multihop FCoE. Unless I’m mistaken, it’s still as common as unicorn tears.

Disclaimer: the examples sprinkled throughout the post are based on my understanding of the functionality of devices that are actually shipping as of late August 2011. Your corrections are most welcome.

More information

FCoE and other storage protocols are described in my Data Center 3.0 for Networking Engineers (recording). The webinar is also available as part of the yearly subscription.

2011-08-30 16:45 GMT - fixed Nexus 6100 FCoE uplink information. Thank you, J Metz!.


  1. Very well done, Ivan. One quick caveat about the Nexus 6100 as of this writing. You are correct that the 6100 can connect to a FCoE target upstream (as long as there is a FC fabric created somewhere, either via MDS, N5k, or even Brocade using FC NPV mode). As of this moment, there is no FCoE "northbound" to another switch, however. This functionality is forthcoming, but I believe it's fair to give people the best information to make decisions at this point in time.

    As soon as this changes, I'll let you know. :)
  2. The funny part is that it wasn't until after I wrote the original article that I realized you came up with the terms SM/DM (I'd lazily assumed that it was part of the standard). I'm not a multi-cast guy, so it took a little bit to figure out why the actual terms were so widely reviled :)

    The article came from my frustration that FCoE is such a broad term that it tells you nothing on how a device can actually fit into a network, yet vendors tend to talk like the small subset of the ways they implement it represents all of FCoE.
  3. Ivan,

    Amazing post. I'm addicted to your blogs. Thanks a lot for sharing your astounding grasp of difficult technologies with the world.

  4. "Vendors tend to talk like the small subset of the ways they implement it represents all of FCoE."

    Tony, that's not really fair. FCoE is a protocol, just like FC is and TCP/IP is. What you're trying to do is force-fit a metaphor onto FCoE that doesn't exist for any other protocol by tying implementation to it. If we took your sentence and applied the various protocol options, would it still hold water:

    "FC is such a broad term that it tells you nothing on how a device can actually fit into a network"

    "TCP/IP is such a broad term that it tells you nothing on how a device can actually fit into a network"

    "InfiniBand is such a broad term that it tells you nothing on how a device can actually fit into a network"

    None of these are fair criticisms of the *protocols* (although there are many, many fair criticisms to be had!) :)

    Frustration is a way for your brain to tell you you're learning something new, and what you're doing is running a grave risk of confusing yourself later on as people take these building blocks and apply them to different implementations that you have not pre-defined.

    Blaming the vendors (and since I've stepped up to the plate and attempted to answer your questions to the best of my ability, by extension you're blaming me for not being clear enough), seems to imply a willingness (or intention) to dissemble that doesn't exist.
  5. Hi J,

    First of all, I do want to apologize. If I sounded like I was being critical of you in particular (I'm a bit critical of Cisco, but not much) it wasn't my intention. I like the work that you do and you FCoE posts have been most helpful.

    "Vendors tend to talk like the small subset of the ways they implement it represents all of FCoE."

    You're right, that's probably not an entirely fair statement. I didn't mean it is a critique, more of a tendency that comes from being familiar with a certain subset of a topic.

    The Juniper/Cisco spat is a good example of that. Juniper talked about needing TRILL, which depending on how you look at it is true or not true. For them, it's true. For Cisco, it's not. You said that TRILL is absolutely not required for FCoE, which is true (but not true if you have non-FCF forwarding bridges, where you really need something other than STP).

    To the outside learner, that's a bit confusing. And confusion only makes FCoE less attractive (I teach Cisco UCS day in and day out, and FCoE doesn't tend to be popular among anyone in any of the classes, even among the FC people).

    So that's why I've been following Ivan's lead in breaking it down. It seems to be helping as well. In Server Load Balancing (ACE, F5, etc.), it's all load balancing, but the way you put a load balancer into a network is different. There's routing mode, one-armed mode, DSR, a few others. It all does the same thing (mostly), but in terms of how you architect a network, it makes a big different. And there are design considerations for each.

    I think it makes sense to do the same thing with FCoE.

    When you consider the difference between FCF and non-FCF switching, it makes much more sense. You did a really good job of explaining FCF-based FCoE switching, my only actual critique was that you didn't explain that there were other ways to do it (and the FCF-, which I agree).

Add comment