In Cisco IOS, when a packet is marked by IOS for ICMP redirect to a better gateway, that packet is being punted to the CPU, right?
It depends on the platform, but it’s going to hurt no matter what.
Regardless of how a networking device forwards packets (in software or hardware) the ICMP redirect is usually not sent from the fast path – the ICMP code has to grab a new packet buffer, create the packet, and send it to the original sender, and particularly the grab a new packet buffer operation is not well-suited to let’s move the packets as fast as possible mentality… unless of course you’ve pre-allocated plenty of buffers for ICMP replies, but even then the CPU cache misses would degrade the performance.
Summary: It might be possible to send ICMP replies from fast packet switching path in software-based packet forwarding platforms. Trying to solve the same problem in packet forwarding hardware is probably an overkill – I would expect those devices to punt offending packets to the CPU.
ICMP redirect is just an indication of potentially better forwarding path, so it can be sent in the background while the original packet has long left the box… at least in theory. I don’t know enough about actual implementations to figure out what’s really going on. Comments highly welcome!
Does that mean that when a packet needs ICMP redirect and it’s punted from a linecard (or ASIC) to the CPU its forwarding becomes totally suboptimal? Or will the CPU punted packet still use CEF forwarding on the CPU?
No idea. Feedback would be highly appreciated!