EBGP Load Balancing with a Multihop EBGP Session

Multihop EBGP sessions are the traditional way to implement EBGP load balancing on parallel links. EBGP session is established between loopback interfaces of adjacent routers (see the next diagram; initial router configurations are included at the bottom of the article) and static routes (or an extra instance of a dynamic routing protocol) are used to achieve connectivity between loopback interfaces (BGP next-hops). The load balancing is an automatic result of the recursive route lookup of BGP next hops.

The following text written by Ivan Pepelnjak in 2009 was originally published on CT3 wiki. That web site became unreachable in early 2019. We retrieved the original text from the Internet Archive, cleaned it up, updated it with recent information if necessary, and republished it on ipSpace.net blog on March 13, 2009

Contents

EBGP session between loopback interfaces is most appropriate in scenarios where all the links between the EBGP neighbors have identical bandwidth. To achieve proportional load balancing across links with different bandwidths, use parallel EBGP sessions in the unequal-bandwidth scenarios.

We’ll illustrate the concepts described in this article with a 2-router testbed shown in the following diagram:

EBGP load balancing testbed

EBGP load balancing testbed

Multihop EBGP Session with Static Routes

The following configuration steps are necessary when you decide to use static routes to provide connectivity between the loopback interfaces (endpoints of the EBGP session):

  • Configure static routes toward the loopback address of EBGP neighbor. The static routes can point to the next-hop (see the configuration of R1) or the physical interface (as configured on E1).
Static route configuration on R1
ip route 10.0.0.23 255.255.255.255 10.0.1.1
ip route 10.0.0.23 255.255.255.255 10.0.1.9
Static route configuration on E1
ip route 10.0.0.1 255.255.255.255 Serial1/0
ip route 10.0.0.1 255.255.255.255 Serial1/3
Static routes pointing to the physical interface shall be used only on point-to-point interfaces.
  • Configure EBGP neighbor. Use the neighbor’s loopback address and specify the source address of the EBGP session with the neighbor address update-source loopback x router configuration command.
BGP neighbor configuration on R1
router bgp 64800
 neighbor 10.0.0.23 remote-as 65000
 neighbor 10.0.0.23 update-source Loopback0
BGP neighbor configuration on E1
router bgp 65000
 neighbor 10.0.0.1 remote-as 64800
 neighbor 10.0.0.1 update-source Loopback0
If you don’t use the neighbor update-source router configuration command, the EBGP session will not be established. When the source interface of the BGP session is not specified, the router uses the IP address of the outgoing link as the source address of the BGP session and the EBGP peer will reset the incoming BGP session coming from an unknown IP address.
  • Configure multihop EBGP session with neighbor ebgp-multihop router configuration command.
EBGP multihop configuration on R1
router bgp 64800
 neighbor 10.0.0.23 ebgp-multihop 2
EBGP multihop configuration on E1
router bgp 65000
 neighbor 10.0.0.1 ebgp-multihop 2
Without the neighbor ebgp-multihop router configuration command, the EBGP session will not be established as Cisco IOS expects a regular EBGP neighbor to be directly connected unless you disable the directly-connected check (see below).

The neighbor ebgp-multihop command makes EBGP sessions vulnerable to TCP session hijacking. To minimize this risk, use the lowest possible TTL value (2) as the maximum hop count parameter in this command. The default TTL value setting (when the neighbor ebgp-multihop is used without the maximum hop count parameter) is 255. For more details on securing BGP routing, read RFC 7454.

After the neighbor ebgp-multihop has been configured, the EBGP session is established and E1 advertises a BGP prefix to R1:

EBGP session on R1
R1#show ip bgp summary | begin Neighbor
Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.0.0.23       4 65000      20      18        6    0    0 00:06:53        1

The show ip bgp neighbor indicates that the neighbor 10.0.0.23 is indeed an EBGP neighbor that is up to two hops away:

EBGP neighbor details on R1
R1#show ip bgp neighbor 10.0.0.23
BGP neighbor is 10.0.0.23, remote AS 65000, external link
  BGP version 4, remote router ID 10.0.0.23
  BGP state = Established, up for 00:07:50

  … deleted …

  Address tracking is enabled, the RIB does have a route to 10.0.0.23
  Connections established 2; dropped 1
  Last reset 00:07:59, due to Peer closed the session
  External BGP neighbor may be up to 2 hops away.
  Transport(tcp) path-mtu-discovery is enabled

  … deleted …

After BGP has converged, the BGP routing table on R1 contains a single copy of the prefix 10.7.1.0/24 advertised by E1.

BGP table on R1
R1#show ip bgp | begin Network
   Network          Next Hop            Metric LocPrf Weight Path
*> 10.7.1.0/24      10.0.0.23                0             0 65000 i 

The IP routing table also contains a single copy of the prefix …

Route to 10.7.1.0/24 in the IP routing table
R1#show ip route 10.7.1.0
Routing entry for 10.7.1.0/24
  Known via "bgp 64800", distance 20, metric 0
  Tag 65000, type external
  Last update from 10.0.0.23 00:12:34 ago
  Routing Descriptor Blocks:
  * 10.0.0.23, from 10.0.0.23, 00:12:34 ago
      Route metric is 0, traffic share count is 1
      AS Hops 1
      Route tag 65000 

… but the CEF table (which contains the results of the recursive lookup) has two equal-cost paths to the prefix:

Multiple entries to 10.7.1.0/24 in the CEF table
R1#show ip cef 10.7.1.0
10.7.1.0/24
  nexthop 10.0.1.1 Serial1/0
  nexthop 10.0.1.9 Serial1/2 

Fast Failover of a Multihop EBGP Session

Cisco IOS release 15.2(4)S and IOS XE release 3.6S added support for multihop BFD checks on multihop EBGP sessions. To use BFD to detect EBGP multihop session failure configure neighbor fall-over bfd multi-hop. If you’re running an earlier Cisco IOS release or if one of the EBGP peers doesn’t support multihop BFD, use BGP fall-over functionality.

BGP fast external failover that is enabled by default on all EBGP sessions does not work on multihop EBGP sessions. Fast external failover is tied to the changes in physical interface status and does not use the IP routing table to check EBGP neighbor reachability.

For example, after both links from R1 to E1 fail, R1 detects the failure only after the BGP keepalive timer (with the default value of 3 minutes) expires.

BGP keepalive timer detects EBGP neighbor failure
16:45:41.127: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0,↩
 changed state to down
16:45:47.619: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/2,↩
 changed state to down
16:48:17.091: %BGP-5-ADJCHANGE: neighbor 10.0.0.23 Down↩
 BGP Notification sent
16:48:17.095: %BGP-3-NOTIFICATION: sent to neighbor 10.0.0.23 4/0↩
 (hold time expired) 0 bytes 

To improve the detection of an EBGP session failure when the EBGP session uses the neighbor ebgp-multihop option, you have to configure fast BGP neighbor loss detection with the neighbor fall-over command. It’s recommended to use a route-map matching only host routes with the neighbor fall-over command to ensure that the router does not interpret a summary route or a default route as a viable path toward the EBGP neighbor. The fast BGP neighbor loss configuration on E1 is shown in the following printout:

Fast BGP neighbor loss detection configuration on E1
router bgp 65000
 neighbor 10.0.0.1 fall-over route-map EBGP_multihop
!
ip prefix-list EBGP_multihop seq 5 permit 0.0.0.0/0 ge 32
!
route-map EBGP_multihop permit 10
 match ip address prefix-list EBGP_multihop 

After the neighbor fall-over has been configured on E1, the EBGP session failure detection improves dramatically (the session failure is detected within a few seconds after the last link toward the neighbor fails).

Fast BGP neighbor loss detection on E1
16:57:11.391: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0,↩
 changed state to down
16:57:13.915: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/3,↩
 changed state to down
16:57:15.231: %BGP-5-ADJCHANGE: neighbor 10.0.0.1 Down Route to peer lost 

Final Router Configurations

Final configuration of E1
service timestamps debug uptime
service timestamps log datetime msec
no service password-encryption
!
hostname E1
!
no aaa new-model
no ip source-route
ip cef
!
archive
 log config
  hidekeys
!
interface Loopback0
 ip address 10.0.0.23 255.255.255.255
!
interface Serial1/0
 ip address 10.0.1.1 255.255.255.252
 encapsulation ppp
 no peer neighbor-route
 keepalive 3
!
interface Serial1/3
 ip address 10.0.1.9 255.255.255.252
 encapsulation ppp
 no peer neighbor-route
 keepalive 3
!
router bgp 65000
 no synchronization
 bgp log-neighbor-changes
 network 10.7.1.0 mask 255.255.255.0
 neighbor 10.0.0.1 remote-as 64800
 neighbor 10.0.0.1 ebgp-multihop 2
 neighbor 10.0.0.1 update-source Loopback0
 neighbor 10.0.0.1 fall-over route-map EBGP_multihop
 no auto-summary
!
ip forward-protocol nd
ip route 10.0.0.1 255.255.255.255 Serial1/0
ip route 10.0.0.1 255.255.255.255 Serial1/3
ip route 10.7.1.0 255.255.255.0 Null0
no ip http server
no ip http secure-server
!
ip prefix-list EBGP_multihop seq 5 permit 0.0.0.0/0 ge 32
!
route-map EBGP_multihop permit 10
 match ip address prefix-list EBGP_multihop
!
control-plane
!
line con 0
 stopbits 1
line aux 0
 stopbits 1
line vty 0 4
 login
!
end 
Final configuration of R1
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname R1
!
no aaa new-model
no ip source-route
ip cef
!
archive
 log config
  hidekeys
!
interface Loopback0
 ip address 10.0.0.1 255.255.255.255
 ip ospf 1 area 0
!
interface FastEthernet0/0
 ip address 10.2.0.1 255.255.255.0
 ip ospf 1 area 0
 duplex half
!
interface Serial1/0
 ip address 10.0.1.2 255.255.255.252
 encapsulation ppp
 no peer neighbor-route
 keepalive 3
 serial restart-delay 0
!
interface Serial1/2
 ip address 10.0.1.10 255.255.255.252
 encapsulation ppp
 no peer neighbor-route
 keepalive 3
 serial restart-delay 0
!
!
router ospf 1
 log-adjacency-changes
 passive-interface default
 no passive-interface FastEthernet0/0
!
router bgp 64800
 no synchronization
 bgp log-neighbor-changes
 neighbor 10.0.0.23 remote-as 65000
 neighbor 10.0.0.23 ebgp-multihop 2
 neighbor 10.0.0.23 update-source Loopback0
 neighbor 10.0.0.23 fall-over route-map EBGP_multihop
 no auto-summary
!
ip forward-protocol nd
ip route 10.0.0.23 255.255.255.255 10.0.1.1
ip route 10.0.0.23 255.255.255.255 10.0.1.9
no ip http server
no ip http secure-server
!
ip prefix-list EBGP_multihop seq 5 permit 0.0.0.0/0 ge 32
!
route-map EBGP_multihop permit 10
 match ip address prefix-list EBGP_multihop
!
control-plane
!
line con 0
 stopbits 1
line aux 0
 stopbits 1
line vty 0 4
 login
!
end 

Single-hop EBGP session between loopback interfaces

IOS releases prior to 12.2(13)T and 12.0(22)S required a multihop EBGP session between loopback interfaces of adjacent routers. Adjacent routers are by definition not using true multihop session, but the neighbor ebgp-multihop router configuration command was still needed to bypass the “directly connected EBGP neighbor” check built into Cisco IOS.

The neighbor disable-connected-check router configuration command disables that check without changing the TTL value in outgoing TCP packets. The fast failover behavior or BFD support remains unchanged: since the neighbor’s IP address is not directly connected the default failover detection does not work. You have to use multihop BFD or the neighbor fall-over router configuration command.

BGP configurations of E1 and R1 where the neighbor ebgp-multihop has been replaced with the neighbor disable-connected-check command are included below:

Configuration of E1
router bgp 65000
 no synchronization
 bgp log-neighbor-changes
 network 10.7.1.0 mask 255.255.255.0
 neighbor 10.0.0.1 remote-as 64800
 neighbor 10.0.0.1 disable-connected-check
 neighbor 10.0.0.1 update-source Loopback0
 neighbor 10.0.0.1 fall-over route-map EBGP_multihop
 no auto-summary
!
ip route 10.0.0.1 255.255.255.255 Serial1/0
ip route 10.0.0.1 255.255.255.255 Serial1/3
ip route 10.7.1.0 255.255.255.0 Null0
!
ip prefix-list EBGP_multihop seq 5 permit 0.0.0.0/0 ge 32
!
route-map EBGP_multihop permit 10
 match ip address prefix-list EBGP_multihop 
Configuration of R1
router bgp 64800
 no synchronization
 bgp log-neighbor-changes
 neighbor 10.0.0.23 remote-as 65000
 neighbor 10.0.0.23 disable-connected-check
 neighbor 10.0.0.23 update-source Loopback0
 neighbor 10.0.0.23 fall-over route-map EBGP_multihop
 no auto-summary
!
ip route 10.0.0.23 255.255.255.255 10.0.1.1
ip route 10.0.0.23 255.255.255.255 10.0.1.9
!
ip prefix-list EBGP_multihop seq 5 permit 0.0.0.0/0 ge 32
!
route-map EBGP_multihop permit 10
 match ip address prefix-list EBGP_multihop 

After the BGP session between E1 and R1 is established, you can use the show ip bgp neighbor command to inspect the details of the EBGP session. The output of this command on R1 confirms that the E1 is an EBGP neighbor that is not directly connected, but R1 nonetheless uses TTL=1 in outgoing TCP packets.

BGP neighbor details on R1
R1#show ip bgp neighbor 10.0.0.23
BGP neighbor is 10.0.0.23,  remote AS 65000, external link
  BGP version 4, remote router ID 10.0.0.23
  BGP state = Established, up for 00:33:08

  … deleted …

  Address tracking is enabled, the RIB does have a route to 10.0.0.23
  Connections established 1; dropped 0
  Last reset never
  External BGP neighbor not directly connected.
  Transport(tcp) path-mtu-discovery is enabled
Connection state is ESTAB, I/O status: 1, unread input bytes: 0
Connection is ECN Disabled, Mininum incoming TTL 0, Outgoing TTL 1
Local host: 10.0.0.1, Local port: 39201
Foreign host: 10.0.0.23, Foreign port: 179
Connection tableid (VRF): 0

  … deleted …

4 comments:

  1. This comment has been removed by a blog administrator.
  2. The comment made by Blogking was a pure product advertisement with no added value.
  3. Hi Ivan,

    a recent comment made by Tony Li[1] regarding eBGP multihop (which I interpret as being averse to it) reminded me I wanted to ask you about this, but never actually did. :) I've been told (and I also remember reading somewhere) that eBGP is a Bad Thing and should be avoided, but I never found much details about why (granted, my google-fu might be low). I stumbled upon this blog post of yours, read the wiki article and found out about the TCP session hijacking issue (and how it can be mitigated). Also, because the neighbour is farther away, I'm thinking the session is (statistically) more likely to be unstable (because there are more devices/links that may experience issues) compared to when the neighbour is directly connected. Besides those, are there any other reasons why eBGP multihop is a Bad Thing?

    Thanks

    [1] http://www.gossamer-threads.com/lists/nanog/users/129697#129697
  4. Let's try to be diplomatic: some very senior networking engineers have strong feelings about certain features (some about multihop EBGP, others about stateful firewalls), undoubtedly because they had to troubleshoot them way too often.

    Like with a number of other advanced features (and sharp knives) you can easily hurt yourself with multihop EBGP if you don't know what you're doing, but it can also be tool that can save the day (admittedly mostly fixing bad designs).
Add comment
Sidebar