Why would FC/FCoE scale better than iSCSI?
During one of the iSCSI/FC/FCoE tweetstorms @stu made an interesting claim: FC scales to thousands of nodes; iSCSI can’t do that.
You know I’m no storage expert, but I fail to see how FC would be inherently (architecturally) better than iSCSI. I would understand someone claiming that existing host or storage iSCSI adapters behave worse than FC/FCoE adapters, but I can’t grasp why properly implemented iSCSI network could not scale.
Am I missing something? Please help me figure this one out. Thank you!
Theoretically iSCSI can scale, but in practice, the management tools have not been put in place. iSCSI is configured on a host level, while FC can be centrally managed. The ecosystem for iSCSI is centered around Microsoft hosts (iSCSI initiator) with storage from Dell (EqualLogic), HP (LeftHand), NetApp and EMC (CLARiiON now VNX) - all of these are midrange products that are not targeted for larger configurations. In talking with many of these vendors, the average configuration for iSCSI tends to be around 20 hosts and it was very rare to find a customer that had deployed 100 servers in a single network. Once again, there is no architectural limitation, but from an operations standpoint it is prohibitive and from what I have seen, no company is putting together the tools to allow simple deployment and management of large scale iSCSI (would probably want to further develop around iSNS). The average FC switch is larger than the typical iSCSI environment, with edge switches going to 80 or more ports and directors to hundreds of ports.
iSCSI scales the same way that creating user account on everyone desktop scales . . . n^((n-1)/2)*. You need a centralized AAA, TE and security mechanism for block disc access. FC provides that.
I have customers who in 18 months have hit the wall on iSCSI deployments cause the disc outgrew the network too fast. Like the early days of VoIP.
* I haven't actually proven that math, but, could do so for a small donation of a bottle of Rye.
I think it's also about control. SAN and FC people like being separate, so that nothing messes up the SAN. That seems to me to be the source of most of the resistance to FCoE. What, share a cable?! How could one possibly troubleshoot in, gasp, someone else's box?! (And the network guys meanwhile worry about FCIP killing their shared WAN link?)
I gather iSCSI is gatewayed. OTOH, in a sense the FCoE FCF forwarding to native FC is too. OTOH, the latter is arguably simpler (strip or add L2 header).
Some people argue that iSCSI is higher overhead. I'd think that depends on the NIC.
Since iSCSI shares a wire with other network traffic, what do you do for QoS? It's the worst case, need it there ASAP but there's a LOT of it. And if it goes lossy on you, or duplicates packets, FC and related SCSI protocols don't generally handle that at all well.
If you have DCB-enabled switches, it would be best to use separate 802.1p priority value for iSCSI traffic, make it lossless (with PFC) and allocate it a fixed bandwidth percentage (with ETS/WDRR).
However, a dedicated network makes it deterministic, that is, performance can be determined absolutely in terms of storage traffic parameters. Note that you still cannot guarantee delivery unless you oversubscribe every element of the network..... which is what FC does at great expense to the customer.