Musings on IP Packet Reordering
During the Comparing Transparent Bridging and IP Routing part of How Networks Really Work webinar I said something along the lines of:
While packets should never be reordered in transit in transparent bridging, there’s no such guarantee in IP networks, and IP applications should tolerate out-of-order packets.
One of my regular readers who designs and builds networks supporting VoIP applications disagreed with that citing numerous real-life examples.
Of course he was right, but let’s get the facts straight first:
- Transparent bridging emulates the original Ethernet cable, and some applications designed to run in that environment expected strict ordering of packets. Some of them (example: IBM SNA) would even reset the session when receiving an out-of-order packet, so we had to take great precautions when implementing various source-route-bridging-over-IP stupidities (RSRB, FST, DLSW, DLSW+…) to preserve the packet ordering.
- IP routing never made such guarantees. It was always understood that packets could arrive out-of-order and that the transport protocol (TCP) or the applications would have to deal with that. Stateless applications (DNS) have no problem whatsoever, and streaming applications could either buffer the packets or ignore out-of-order packets resulting in lower voice- or video quality.
Nobody ever claimed packet reordering is NOT harmful, although (IIRC) people who know better told me it’s no longer a big problem with decent TCP stack, and I encountered a not-insignificant amount of reordered packets when I had to troubleshoot traffic traversing the Internet.
However, reality and laws of physics (and networking) have never stopped application developers from assuming that real life behaves in exactly the same way as their loopback interface, and I blame their managers for not exposing them to that fact during testing- and acceptance procedures. After all, you could easily simulate a real-life network behavior with tc or use a CLI wrapper around tc like Comcast if you can’t be bothered to figure out its arcane options.
I also know that a lot of networking engineers get pushed into designing or implementing networks that support such delusions, but if that’s the case, please don’t claim that IP has to work the way you made it work, it’s just that you are unfortunate enough to have to support something with broken expectations that deviate from how IP was always supposed to work.
On the other hand, it might turn out that there’s a good use case behind the weird requirements, and that you need something different from vanilla IP routing. No problem, fork IP, and go your own way. For example, there’s TCP-over-lossless-transport (apparently not such a good idea), then there’s Deterministic Networking (DETNET), and I’m sure there are other similar examples… but should we still call them IP routing? I don’t know.
1) Apart from a bad implementation or configuration or a bug of the hashing algorithm governing the load-balancing, can these days packet reordering still be introduced within an high-end router while traversing it's fabric layers from the input to the output port/card ? I remember witnessing it happening on some juniper boxes during the early 2000s after some very deep testing for an R&D European project.
2) there's no such an alarm/counter on a (stateless by definition) router as 'number of out of sequence packets' .... since you'd need to sequence every flow's packet (according to the forwarding chipset's configured flow hashing algorithm granularity...) coming in and then work out on a per-flow basis if there's some reordering been introduced while sending it out on a port....
3) To my knowledge, very few providers stress-test their routers themselves in their labs the way you should to be able to spot such issues. And this should be tested every time you change code or card too...
4) could therefore packet reordering be an Internet's physiological issue (sort of background noise) that is fixable but at a cost so high that no one really cares unless it becomes a real problem for a complaining customers of yours (such as when you configure a brand new lag towards a customer and forget to change the load-balancing default algorithm from packet based to flow based as witnessed recently @ an IX) ?
Packet reordering could be introduced anywhere, it's just the question of whether it's a rare event or a continuous behavior. The simplest way (I'm aware of) would be remarking out-of-contract packets and then using a different queue for them.
I would strongly suggest to go with #4 in your list, and whenever a customer complains do a thorough evaluation whether it's worth the effort to fix it balancing immediate costs + technical debt against potential profit (NOT revenue) made on that customer.
Finally, having a recent product using per-packet load balancing as a default LAG algorithm deserves a Triple Facepalm. If you feel like sharing who the offending vendor is, you know how to reach me.
I was involved in many issue investigations caused by the reordering during last 10 years. Most of them were caused by networking equipment coming from major vendors - stateful devices usually are the culprit.
Parallel processing not handled correctly especially during the phase of FW session establishment (packets are buffered during this period and the process of retrieving them from the buffer is challenging from the reorder handling perspective) is one of the common root cause example.
I spent some time investigating these issues because of the performance degradation reported in the network (udp protocol failures, not-so-decent tcp implementation perf degradation, multicast audio streams affected, etc).
We may think about these issues as a niche cases but they are real and we can simple change the legacy protocol.
If it comes to vendors. They do some testing for typical protocols and scenarios (80% of use cases). For example multicast is poorly tested particularly on cheaper boxes. I would assume that cheap boxes are usually affected by reordering to certain degree.
Every vendor when you report re-ordering issues denies it because this problem is generally solved (they mark packets incoming to the box and they transmit them preserving order). But as always we have corner cases e.g. what happens when we have not created the firewall state on the devices and we dot want to drop packet?
It's rather complex topic.
@BOGDAN GOLAB Bogdan, could you provide more details on the following quoted statement of yours? "Every vendor when you report re-ordering issues denies it because this problem is generally solved (they mark packets incoming to the box and they transmit them preserving order)" I am pretty sure high-end router vendors only just reorder cells belonging to the same packet after having traversed the fabric planes as in that case there's state to preserve where the state is an ordered packet. I can't see how they can do what you just described to be honest.
Not sure if I want to discus this in public.
I think about two top vendors at least (both denied at the beginning and provided some internal architecture details to prove this - unfortunately it did not cover some corner cases, both delivered fixes - but it took time ).
Talking mostly about Firewalls as mentioned earlier
This is not high end equipment - rather high end branch. I do not know high end architectures so I cannot compare the solutions
Please review some patents related to ensuring packet order by NPE.