A while ago I described why some storage vendors require end-to-end layer-2 connectivity for iSCSI replication.
TL&DR version: they were too lazy to implement iSCSI checksums and rely on Ethernet checksums because TCP/IP checksums are not good enough.
It turns out even Ethernet checksums fail every now and then.
2015-12-06: I misunderstood the main technical argument in Evan’s post. The real problem is that switches recalculate CRC, so the Ethernet CRC is no longer end-to-end protection mechanism.
2015-11-15: Fixed: TCP/IP checksums are not XORs (thanks to Andrew Yourtchenko).
TCP and IP checksums are simple
XOR operations , and we know they’re weak. As Evan Jones explained in his blog, you might expect that one in ~65000 corrupt packets won’t be detected, which combined with pretty low error rates we see on Ethernet these days might be good enough… or not, if you’re Twitter and dealing with petabytes of traffic.
Ethernet CRC is supposed to save the day. After all, a switch receiving a packet checks the CRC regardless of whether the packet is subsequently bridged or routed. Ethernet CRC should reliably detect transmission errors, and the TCP/IP checksums should detect extremely rare intra-device data corruption errors… or so the theory goes.
In practice, there’s a gap between theory and practice: cut-through switches (becoming yet again ever more popular due to reduced latency) don’t check the CRC
. Even worse, they slap the correct CRC on corrupted data, making it impossible to detect the corruption further down the line (more details in an excellent blog post by John Harrington),
Evan’s conclusion: if you care about data integrity, implement application-level checksum, preferably using CRC32C, which is implemented in hardware on recent CPUs.
Also note: Stretched VLAN is not a data protection feature for your iSCSI network. If the iSCSI or NFS solution you’re using doesn’t support application-level checksums, your data is at risk no matter what.
Finally, how many application-level protocols apart from SSL/TLS and iSCSI (when implemented) implement an application-level checksum? Please write a comment!