IP packet delivery confirmation

Thomas wanted to check whether the IP traffic is actually delivered to a remote site and sent me the following question:

I would like to know whether the packets I sent from site A to site B have been received. I don't want to create test traffic using ip sla, I would like to know that the production traffic has been delivered. I could use ACL counters but I'm running a full mesh of tens of sites. Ipanema does this very well, but I'm surprised that this doesn’t exist on Cisco IOS.

Short answer: that’s not how Internet works.

One of the fundamental architectural principles of Internet is smart hosts with simple core. As stated in RFC 1958 (section 2.3):

[...] any network, however carefully designed, will be subject to failures of transmission at some statistically determined rate. The best way to cope with this is to accept it, and give responsibility for the integrity of communication to the end systems.

The routers are by definition simple devices that forward individual packets; the transport protocol implemented in the end hosts should take care of reliable data delivery. Quoting from RFC 3439 (section 2.1):

[...] end-to-end protocol design should not rely on the maintenance of state (i.e., information about the state of the end-to-end communication) inside the network. Such state should be maintained only in the end points, in such a way that the state can only be destroyed when the end point itself breaks.

The benefits of this architecture are clear: almost everything (including voice and streaming video) is transported across IP today due to IP's cost efficiency.

Introducing state in the network (example: tracking packets) always increases complexity and reduces performance. Read RFC 3439 for more details.

There have been technologies that maintained per-session state within the network: X.25, SNA and ATM-to-the-desktop are prime examples. Their current popularity reflects their performance, cost efficiency and scalability. Fibre Channel also ensures (almost) lossless transport across the fabric. Price-per-port comparison between 8Gbps FC and 10GE (non-DCB/FCoE) switches should give you an idea of the relative complexity of the two approaches (hint: to get the real picture, do the comparison without the transceivers).

Finally, the application acceleration solutions almost always replace end-to-end TCP sessions with their own protocol, terminating TCP sessions locally (and thus maintaining per-session state) and using a proprietary protocol between individual sites.

Monitoring end-to-end packet delivery performance in an application acceleration solution is thus trivial – you could do the same thing by monitoring the tcpRetransSegs variable in TCP MIBs on individual hosts.


  1. This post was outstandingly great. Thanks a lot.

    First time I've heard a router (Cisco IOS) compared to a network analysis tool (ARX, Ipanema).

    That's really the network analysis tool's job(ie ipanema, ARX, Infinistream). Would anyone really want a wan/lan router that has 80TB RAID? Seems like too much.
  2. But then again, we've been violating both the End-to-end and Simplicity principles for years - NAT, firewalls, load balancers, WAN accelerators, even LAG - all either peek inside the packet beyond the header or maintain state (intelligence) inside the network.

    The result is that the Transport layer is no longer end-to-end, it is at the mercy of various "black boxes" which have become control points for the Internet.

    Maybe its because of such pervasive network-level hardware dependence that other Transport layer protocols (SCTP) have not been able to gain any traction.
Add comment