Category: CEF

In "border cases" you might find interesting CEF adjacencies in your CEF adjacency table (displayed with show ip cef adjacency). Most common one is the glean adjacency used for directly connected routes (this adjacency type is a placeholder that indicates the router it should perform the ARP table lookup and send the packet to directly connected neighbor). Discard, Drop, Noroute and Null adjacencies are obvious, the "weird" one is the Punt adjacency, which indicates that the router cannot CEF-switch the packet toward the destination (due to a feature being used that is not yet supported by CEF), thus the packet is punted to the next switching method (fast switching and ultimately process switching).

In designs with very low number of IP hosts, no per-destination load-sharing algorithm will work adequately. Consider, for example, an extranet design where a large number of IP hosts are NAT-ed to a single IP address which then accesses a single remote server.

In this design, all the traffic flows between a single pair of IP addresses, making per-destination load-sharing unusable.

Cisco Express Forwarding (CEF) can perform per-packet or per-destination (actually source/destination IP address pair) load-sharing with no performance degradation (without CEF, per-packet load-sharing requires process switching). Even though there is no performance impact on the router, per-packet load sharing will almost always result in out-of-order packets. The packet reordering might degrade TCP throughput in high-speed environments (in low-speed/few-flows scenarios, per-packet load-sharing actually improves the per-flow throughput) or severely impact applications that cannot survive out-of-order packet delivery, such as Fast Sequenced Transport for SNA over IP or voice/video streams.

To configure per-packet load-sharing, use the ip load-sharing per-packet interface configuration command (default is per-destination). This command has to be configured on all outgoing interfaces over which the traffic is load-shared.

The switch between the load-sharing modes is not immediate; sometimes you have to wait a few seconds for the ip load-sharing command to take effect, worst case a manual clearing of the CEF table (clear ip cef address) is required.

I've got an excellent question recently: Which switching path is used in Zone-based firewalls when a packet is dropped? As usual, IOS documentation was not very helpful (which is understandable as the answer might depend on hardware platform, interface encapsulation, other features configured on the router etc.). However, there is a great tool to use - the show interface stats command.

In environments with a low number of IP hosts you have to fine-tune the CEF load-sharing algorithm to ensure that the traffic is spread between all parallel paths. A typical scenario is a primary-backup data center setup with pairs of replicating servers, as shown in the figure below.

In these cases, you have to try different values of seed parameter of the CEF universal algorithm.

According to the Cisco IOS documentation, you can select between the original and the universal CEF load sharing algorithm with the ip cef load-sharing algorithm name parameter global configuration command (we'll leave the tunnel algorithm aside for the moment). Of course, they don't tell you what you select.

The original algorithm used only the source and destination IP addresses to get the 4-bit hash entry (see the CEF Load Sharing Details for more information), which could result in suboptimal network utilization in some border cases (if anyone wants to know why, leave me a comment). The universal algorithm adds a router-specific value to the hash function, ensuring that the same source-destination pair will hash into a different 4-bit value on different boxes. If you really want to fine-tune the hash function, you can even specify the value to be added with the last option of the ip cef load-sharing algorithm command.

I had to investigate the details of CEF load sharing for one of my upcoming article and found (yet again) that the details are rather undocumented in official documentation. So, this is how it works (in case you ever need to know):

• For every CEF entry (IP route) where there are multiple paths to the destination, the router creates a 16-row hash table, populating the entries with pointers to individual paths. The hash table can be inspected with the show ip cef prefix internal command.
• The load balancing ratio is approxiated by number of entries in the hash table belonging to each path. If you have unequal-cost load balancing (EIGRP based on composite metrics and MPLS TE tunnels based on requested bandwidth), individual paths will be associated with different number of rows.
• If you configure per-destination load balancing, the source and destination IP address in the incoming IP packet are hashed into a 4-bit value that selects the outgoing path in the CEF has table.