Category: load balancing

Establishing parallel EBGP sessions across parallel links between two edge routers (EBGP peers) – as displayed in the diagram below – is the most versatile form of EBGP load balancing. It does not require static routing or extra routing protocol (like the design running EBGP between routers’ loopback interfaces), device-specific tricks like configuring the same IP address on multiple interfaces) or specific layer-2 encapsulation (like Ethernet LAG or Multilink PPP).

It even allows proportional load-balancing across unequal-bandwidth links and combinations of various layer-2 technologies (for example, load-balancing between a serial line and an Ethernet interface). The only drawback of this design is the increased size of the BGP table, as every BGP prefix is received from the EBGP neighbor twice.

EIGRP was always described as the only routing protocol that can do unequal-cost load sharing. As it turns out, BGP is another one (although it's way more limited than EIGRP). For example, if you have two links into a neighbor AS, you can load-share across them proportionally to their bandwidth.

EBGP load balancing was introduced with the BGP 4 Multipath Support feature in IOS release 11.2. Initially, EBGP supported up to six maximum paths; IOS release 12.0(S) increased that value to 8, IOS release 12.3T to 16 and 12.2S (including 12.2SRC) to 16.

Unequal load-sharing with static routes is almost impossible as there is no configuration command to assign non-default traffic share count to a static route. For example, if you configure two default routes, one pointing to a low-speed interface and another one pointing to a high-speed interface, there is no mechanism to force majority of the traffic onto the high-speed link (IOS ignores interface bandwidth when calculating load sharing ratios).

You can, howerer, use a workaround: if you configure multiple routes for the same prefix pointing to the same interface, that interface will attract proportionally more outbound traffic.

One of the most commonly asked load-sharing-related questions is “can I load-share traffic across unequal-cost links?”. In general, the answer is no. In order to load-share the traffic, you need more than one path to the destination and the only way to get multiple routes toward a destination in the IP routing table is to make them equal-cost (the only notable exception being EIGRP that supports unequal-cost load-sharing with the variance parameter).

There are, however, two cases where you can force unequal traffic split across equal-cost paths toward a destination: when using inter-AS BGP with the link bandwidth parameter, and when using unequal-bandwidth traffic-engineering tunnels.

In designs with very low number of IP hosts, no per-destination load-sharing algorithm will work adequately. Consider, for example, an extranet design where a large number of IP hosts are NAT-ed to a single IP address which then accesses a single remote server.

In this design, all the traffic flows between a single pair of IP addresses, making per-destination load-sharing unusable.

Cisco Express Forwarding (CEF) can perform per-packet or per-destination (actually source/destination IP address pair) load-sharing with no performance degradation (without CEF, per-packet load-sharing requires process switching). Even though there is no performance impact on the router, per-packet load sharing will almost always result in out-of-order packets. The packet reordering might degrade TCP throughput in high-speed environments (in low-speed/few-flows scenarios, per-packet load-sharing actually improves the per-flow throughput) or severely impact applications that cannot survive out-of-order packet delivery, such as Fast Sequenced Transport for SNA over IP or voice/video streams.

To configure per-packet load-sharing, use the ip load-sharing per-packet interface configuration command (default is per-destination). This command has to be configured on all outgoing interfaces over which the traffic is load-shared.

The switch between the load-sharing modes is not immediate; sometimes you have to wait a few seconds for the ip load-sharing command to take effect, worst case a manual clearing of the CEF table (clear ip cef address) is required.

In environments with a low number of IP hosts you have to fine-tune the CEF load-sharing algorithm to ensure that the traffic is spread between all parallel paths. A typical scenario is a primary-backup data center setup with pairs of replicating servers, as shown in the figure below.

In these cases, you have to try different values of seed parameter of the CEF universal algorithm.

According to the Cisco IOS documentation, you can select between the original and the universal CEF load sharing algorithm with the ip cef load-sharing algorithm name parameter global configuration command (we'll leave the tunnel algorithm aside for the moment). Of course, they don't tell you what you select.

The original algorithm used only the source and destination IP addresses to get the 4-bit hash entry (see the CEF Load Sharing Details for more information), which could result in suboptimal network utilization in some border cases (if anyone wants to know why, leave me a comment). The universal algorithm adds a router-specific value to the hash function, ensuring that the same source-destination pair will hash into a different 4-bit value on different boxes. If you really want to fine-tune the hash function, you can even specify the value to be added with the last option of the ip cef load-sharing algorithm command.

I had to investigate the details of CEF load sharing for one of my upcoming article and found (yet again) that the details are rather undocumented in official documentation. So, this is how it works (in case you ever need to know):

• For every CEF entry (IP route) where there are multiple paths to the destination, the router creates a 16-row hash table, populating the entries with pointers to individual paths. The hash table can be inspected with the show ip cef prefix internal command.
• The load balancing ratio is approxiated by number of entries in the hash table belonging to each path. If you have unequal-cost load balancing (EIGRP based on composite metrics and MPLS TE tunnels based on requested bandwidth), individual paths will be associated with different number of rows.
• If you configure per-destination load balancing, the source and destination IP address in the incoming IP packet are hashed into a 4-bit value that selects the outgoing path in the CEF has table.