J Michel Metz brought out an interesting aspect of the dense/sparse mode FCoE design dilemma in a comment to my FCoE over Trill ... this time from Juniper post: FC-focused troubleshooting. I have to mention that he happens to be working for a company that has the only dense-mode FCoE solution, but the comment does stand on its own.
Before reading this post you might want to read the definition of dense- and sparse-mode FCoE and a few more technical details.
Let’s try to look at his comment from a router jockey perspective. Traditional SAN network is a pure routed network. Every FC switch is actually a layer-3 forwarding device. Supposedly there are all sorts of FC monitoring and troubleshooting mechanisms that make life of a SAN administrator easier (or so I am led to believe by FC fans whenever I mention iSCSI). An approximate equivalent in the IP world would be a purely router-based network with, for example, SONET/SDH links between routers (I wanted to say leased lines but I guess quite a few people would stop reading thinking this is another one of my last-millennium rants).
Replacing FC with dense-mode FCoE is like replacing OC-12/STM-4 with DWDM+GE in your router network. The gear gets a bit cheaper, you lose a bit of the functionality (remote alarms, for example), but you still control the whole network. You have hop-by-hop visibility and can do traceroute-like troubleshooting; you can also see all interface counters and are able to evaluate the reliability of every link. In a nutshell, apart from reducing the costs of the L1/L2 technology (and gaining some extra bandwidth in the process), not much has changed and all your troubleshooting tools and processes still work.
The drawbacks of dense-mode FCoE? Nothing has changed. You still have a SAN network we all love to hate, but it’s using 10GE instead of 8G FC. Every LAN switch is a fiber channel switch (exploding your FC domain), FSPF is running everywhere, and every topology change causes two SPF runs (one for FSPF, one for IS-IS if you run TRILL/FabricPath or OSPF if you don’t believe in flat-earth myths). Unless your FC gear supports the FC standards to their maximum limits, you will have to implement N_port proxy (with NPIV) on the FCoE/FC edge to protect the poor innocent FC switches that have never seen planets larger than a few switches. You do know the theoretical limit is 239 switches per SAN network (including all your LAN FCoE gear), right?
Sparse-mode FCoE with FIP-snooping switches is like replacing your SONET/SDH infrastructure with VPLS. You get a misty blob between your routers that you cannot troubleshoot with your existing tools. Interface counters are no longer relevant, because you see only half of the truth on the access link (inbound, but not outbound errors) and have zero visibility into transit links. Hop-by-hop troubleshooting no longer gives exact answers, as the whole blob that sits in the middle of your network (VPLS cloud) behaves like a single hop. Decisions made by routing protocols are somewhat useless, as you never know how many hops there are in the VPLS cloud, what its true bandwidth and latency is, and thus what the cost of the VPLS subnet should be. In fact, routing protocols change from path-computation tools into reachability-distribution tools. Did I mention that you can’t control the convergence speed anymore because most of the failures happen within the blob that you have no influence on?
If you were ever forced to make a migration from a router-only WAN network with point-to-point links to a VPN-based network (be it Frame Relay, L3 MPLS/VPN, VPLS or something-over-Internet), you’re probably acutely aware of the frustrations caused by the loss of control and troubleshooting capabilities. That’s how your SAN administrators might feel if you implement sparse-mode FCoE. The choice is yours.
If you want to learn more about modern data center architectures, including FCoE, Data Center Bridging and TRILL, buy a recording of my Data Center 3.0 for Networking Engineers webinar. To learn more about benefits and drawbacks of individual WAN VPN solutions, consider my Choose the Optimal VPN Service webinar. Both webinars are also part of the yearly subscription package.