Was anyone trying to sell you the “wonderful” idea of running FCoE between Data Centers instead of FC-over-DWDM or FCIP? Sounds great ... until you figure out it won’t work. Ever ... or at least until switch vendors drastically increase interface buffers on the 10GE ports.
FCoE requires lossless Ethernet between its “routers” (Fiber Channel Forwarders – see Multihop FCoE 101 for more details), which can only be provided with Data Center Bridging (DCB) standards, specifically Priority Flow Control (PFC). However, if you want to have lossless Ethernet between two points, every layer-2 (or higher) device in the path has to support DCB, which probably rules out any existing layer-2+ solution (including Carrier Ethernet, pseudowires, VPLS or OTV). The only option is thus bridging over dark fiber or a DWDM wavelength.
If you’ve been working with long-distance Fiber Channel, you know that the end-to-end throughput decreases with distance and number of buffers the switches have due to the Fiber Channel flow control mechanism (BB_credits). BB_credits are a very safe mechanism: if the transmission latency is too large, the confirmations won’t arrive in time and the transmitter will stop. Worst case, you’ll get dismal throughput.
The Priority Flow Control used by lossless Ethernet is the exact opposite of BB_credits: as the receiver starts running out of buffers, it has to send a PAUSE frame to stop the sender. Obviously the PAUSE frame has to be sent soon enough so that the packets sent before the transmitter will receive the PAUSE frame will not overload the input buffers.
Cisco published a detailed white paper describing the limitations of PFC. Short summary: if you want to run PFC across a 10-km link, you need to stop the transmitter while you still have 350K free space in input buffers (the total amount of per-interface buffers space is 500K). PFC implementation in Nexus 5000 is therefore limited to 300 meters (and the distance is not configurable).
With FCoE out of the picture, we’re left with the will-known IP-based options if you can’t get a dedicated wavelength for your Fiber Channel SAN: FCIP, iSCSI or NFS. Told you iSCSI is better than FCoE, did I not?
- Data Center 3.0 for Networking Engineers webinar (buy a recording or yearly subscription) describes numerous Data Center technologies, including DCB and FCoE.
- Read my Data Center Bridging posts to get more information on DCB and PFC
- I also wrote numerous posts covering FCoE architecture and design caveats.