Building Network Automation Solutions
6 week online course starting in September 2017

Firewalls kill TCP performance when faced with out-of-order packets

In my discussion of per-packet versus per-destination load sharing, I've relied on the "accepted wisdom" that out-of-order TCP packets reduce session performance (as a side note, out-of-order UDP packets are a true performance killer; just try running NFS with out-of-order packets).

Today I've discovered another huge show-stopper: stateful firewalls (read: almost everything in use today) might just drop out-of-order packets, resulting in TCP timeouts and retransmissions (and repeated timeouts will totally wreck the session throughput). Here's how Cisco devices handle this problem:


  1. A similar problem is IP fragmentation (although that should not happen too often if MTU discovery is working properly), which is handled with the Virtual Fragmentation Reassembly feature introduced in IOS release 12.3(8)T.

  2. But problem really is not caused by load-sharing. It is because many things aren't prepared for out ouf order packets and it can just happen without load sharing, because that is how Internet works. You can fix your configuration by using per-destination, but it won't fix remote locations.

  3. As I wrote in a comment to another post, I really cannot see packets getting reordered unless you use per-packet load sharing or weird QoS setup that treats packets from the same flow differently.

    If there's something I'm missing, I would be glad to hear about it.

  4. I know this is an old post, but what if I'm having an out-of-order and retransmission packets flood detected by both N5k-s and Wireshark, and I can see checkpoint is dropping them... could this have something to do with the traffic towards the FW passing through the same physical link various times due to the L2 architecture?

    1. Well, it shouldn't ;) Obviously something in the forwarding path is reordering packets (probably due to per-packet ECMP/LAG instead of per-session or per-address ECMP/LAG). Capture traffic at various points along the forwarding path to figure out where they get reordered.


You don't have to log in to post a comment, but please do provide your real name/URL. Anonymous comments might get deleted.