Numerous network devices can combine static routes with Bidirectional Forwarding Detection (BFD) and remove them from the routing table if the BFD session with the next-hop router fails. Unfortunately, that works only if the upstream network supports BFD on its customer-facing interfaces¹; we need a more generic solution that does not rely on the functionality of the upstream router².

Cisco IOS includes Enhanced Object Tracking functionality, which, together with Reliable Static Routing Using Object Tracking, solves the “Is the next hop reachable?” problem without relying on the adjacent router’s cooperation.

Enhanced Object Tracking introduces a generic track object that can track the state of an interface (layer-2 or layer-3 state), the presence or metric of an IP route, the state of an SLA measurement, or even the availability of Mobile IP home agent or GPRS nodes. You can also combine various track objects (including weighing them) into a compound object.

The Reliable Static Routing Using Object Tracking feature ties a track object to a static route – whenever the track object’s state is down, the static route is removed from the routing table; precisely what you would need to support reliable multi-homing. To configure a static route based on the state of the next-hop router, you need to:

• Configure an ip sla (previously known as Response Time Reporter – rtr) object pinging the next-hop router on the primary Internet link. The polling frequency you specify (in seconds) depends on the reliability requirements, but don’t exaggerate. Anything below a few seconds would unnecessarily burden the next-hop router (as you might not be the only one tracking its availability).

ip sla 100
 icmp-echo 172.16.1.2 source-interface GigabitEthernet0/2
 threshold 500
 timeout 1000
 frequency 3
ip sla schedule 100 life forever start-time now

• Create a track object monitoring the reachability of the SLA target.
• As you probably don’t want to respond to a single lost ICMP packet, you should use the delay option of the track object to specify how long the next-hop router should remain unreachable before it’s declared lost.
• The down delay should be at least three times the SLA polling frequency
• To deal with intermittent connectivity, the up delay should be even longer than the down delay. For example, a router can temporarily respond to pings while it boots.

track 100 ip sla 100 reachability
 delay down 10 up 20

• After configuring the track object, attach it to the primary static default route to ensure that the default route is removed if the next-hop router is not reachable:

ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/2 172.16.1.2 10 name ISP_A track 100
ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/3 172.17.3.2 251 name ISP_B_FB

You can check the proper operation of the reliable static routing with the show ip route command. The following listings display:

• The IP routing table on the GW router when the primary next-hop router is available
• The modified state of the IP routing table after the primary next-hop router failure.

gw#show ip route | begin Gateway
Gateway of last resort is 172.16.1.2 to network 0.0.0.0

S*    0.0.0.0/0 [10/0] via 172.16.1.2, GigabitEthernet0/2
      10.0.0.0/32 is subnetted, 1 subnets
C        10.0.0.1 is directly connected, Loopback0
      172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks
C        172.16.1.0/30 is directly connected, GigabitEthernet0/2
L        172.16.1.1/32 is directly connected, GigabitEthernet0/2
      172.17.0.0/16 is variably subnetted, 2 subnets, 2 masks
C        172.17.3.0/30 is directly connected, GigabitEthernet0/3
L        172.17.3.1/32 is directly connected, GigabitEthernet0/3
      192.168.0.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.0.0/24 is directly connected, GigabitEthernet0/1
L        192.168.0.1/32 is directly connected, GigabitEthernet0/1

gw#show ip route | begin Gateway
Gateway of last resort is 172.17.3.2 to network 0.0.0.0

S*    0.0.0.0/0 [251/0] via 172.17.3.2, GigabitEthernet0/3
      10.0.0.0/32 is subnetted, 1 subnets
C        10.0.0.1 is directly connected, Loopback0
      172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks
C        172.16.1.0/30 is directly connected, GigabitEthernet0/2
L        172.16.1.1/32 is directly connected, GigabitEthernet0/2
      172.17.0.0/16 is variably subnetted, 2 subnets, 2 masks
C        172.17.3.0/30 is directly connected, GigabitEthernet0/3
L        172.17.3.1/32 is directly connected, GigabitEthernet0/3
      192.168.0.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.0.0/24 is directly connected, GigabitEthernet0/1
L        192.168.0.1/32 is directly connected, GigabitEthernet0/1

Revision History

2025-03-31

• Mentioned BFD as an alternative to IP SLA
• Mentioned RFC 9747 as a potential BFD-based solution if the upstream router doesn’t want to participate in BFD (HT: Erik Auerswald)
• Recreated the router configurations and printouts with IOSv release 15.6(1)T.
• Added the command to shut down the PE_A interface when using netlab

Reliable static routes silently appear or disappear from the IP routing table based on the state of the attached track object; the only way to monitor their state is to use the show ip route track-table command.

GW#show ip route track-table
 ip route 0.0.0.0 0.0.0.0 Serial0/0/0 10 name ISP_A track 100 state is [down]

However, state tracking generates logging messages, and you can use the debug track command to get even more details.

GW#debug track
06:49:44: Track: 100 Down change delayed for 10 secs
06:49:54: Track: 100 Change #24 ip sla 100, reachability Up->Down
06:49:54: %TRACK-6-STATE: 100 ip sla 100 reachability Up -> Down
06:50:24: Track: 100 Up change delayed for 20 secs
06:50:34: Track: 100 Up change delay cancelled
06:58:59: Track: 100 Up change delayed for 20 secs
06:59:19: Track: 100 Change #25 ip sla 100, reachability Down -> Up
06:59:19: %TRACK-6-STATE: 100 ip sla 100 reachability Down -> Up

Suppose you need more than the logging messages the track objects generate. In that case, you can use the Embedded Event Manager with the event track configuration command to trigger EEM applets (or TCL scripts) whenever a track object’s state changes.

For example, we’ll define two EEM applets: one triggered on the down change, and another triggered on the up change. We’ll use a simple syslog msg action; you can add other actions, like emailing the network manager or even reconfiguring the router (the complete router configuration is available on GitHub).

The following printouts show a sample EEM configuration and the printouts it generates.

event manager applet ISP_A_down
 event track 100 state down
 action 1.0 syslog msg "ping to 172.16.1.2 from GigabitEthernet 0/2 failed"
event manager applet ISP_A_up
 event track 100 state up
 action 1.0 syslog msg "172.16.1.2 is reachable"

00:38:52: %TRACK-6-STATE: 100 ip sla 100 reachability Up -> Down
00:38:52: %HA_EM-6-LOG: ISP_A_down: ping to 172.16.1.2 from GigabitEthernet 0/2 failed
00:39:22: %TRACK-6-STATE: 100 ip sla 100 reachability Down -> Up
00:39:22: %HA_EM-6-LOG: ISP_A_up: 172.16.1.2 is reachable

Revision History

2025-03-31

• Recreated the router configurations and printouts with IOSv release 15.6(1)T.
• Object tracking generates logging messages in newer IOS releases

After you’ve successfully implemented the tracking of the primary next-hop router’s availability, you might be tempted to improve the solution to track end-to-end connectivity through ISP A and switch to the backup ISP whenever your central site is not reachable through the primary ISP. In theory, the required configuration change should be minimal – you only have to change the destination IP address in the IP SLA definition:

hostname GW
!
ip sla 100
 icmp-echo 172.18.0.6 source-interface GigabitEthernet0/2
 threshold 500
 timeout 1000
 frequency 3
ip sla schedule 100 life forever start-time now

Unfortunately, there’s a serious problem with this setup when the path between GW and PE_A fails in a way that is not detected by the GW router (for example, there’s a problem in an intermediate layer-2 switch):

• IP SLA fails, and the default route to PE_A is removed
• GW installs the default route to PE_B
• Pings are now sent from an IP address belonging to the ISP-A uplink onto a path going through the ISP-B.
• The redirected pings go through a NAT translation.
• You’ll get an oscillating default route if the reverse NAT works and the ICMP responses reach the IP SLA measurement code. Otherwise, the IP SLA test will keep failing, and the default route to ISP A will not be installed even when the connectivity with PE A is restored.

To fix this problem, you have to configure a local policy routing (as the ip sla packets originate within the router, they are only affected by the ip local policy) that matches ICMP packets being sent from the GigabitEthernet0/2 interface (based on their IP address; the PingISP_A access list) and forces them to be sent out through the correct interface toward the expected next hop:

ip local policy route-map LocalPolicy
!
ip access-list extended PingISP_A
 permit icmp host 172.16.1.1 host 172.18.0.6
!
route-map LocalPolicy permit 10
 match ip address PingISP_A
 set ip next-hop 172.16.1.2
 set interface GigabitEthernet0/2

Coping with the Central Site Failure

You can extend the concepts presented in this section even further if you want to. For example, if the central site is not reachable through either ISP (it might be down), retaining ISP A as the primary ISP could make more sense. You would thus need to:

• Track the central site’s availability through both ISPs.
• Configure a reliable static default route for both ISPs (using a higher administrative distance on the backup one).
• Add a third (last-resort) default route pointing to ISP A.
• Add an even worse default route pointing to ISP B (in case the interface toward ISP A fails).

The relevant parts of the router configuration are included in the following listing (the complete configuration is on GitHub), and its interpretation is left as an exercise for the reader.

hostname gw
!
ip dhcp pool LAN
 network 192.168.0.0 255.255.255.0
 default-router 192.168.0.1 
!
track 100 ip sla 100 reachability
 delay down 10 up 20
!
track 101 ip sla 101 reachability
 delay down 10 up 20
!
interface GigabitEthernet0/1
 description gw -> [h1,h2] [stub]
 ip address 192.168.0.1 255.255.255.0
 ip nat inside
!
interface GigabitEthernet0/2
 description gw -> pe_a
 ip address 172.16.1.1 255.255.255.252
 ip nat outside
!
interface GigabitEthernet0/3
 description gw -> pe_b
 ip address 172.17.3.1 255.255.255.252
 ip nat outside
!
ip local policy route-map LocalPolicy
!
ip nat inside source route-map ISP_A interface GigabitEthernet0/2 overload
ip nat inside source route-map ISP_B interface GigabitEthernet0/3 overload
ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/2 172.16.1.2 10 name ISP_A track 100
ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/3 172.17.3.2 11 name ISP_B track 101
ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/2 172.16.1.2 250 name ISP_A_FB
ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/3 172.17.3.2 251 name ISP_B_FB
!
ip access-list extended PingISP_A
 permit icmp host 172.16.1.1 any
ip access-list extended PingISP_B
 permit icmp host 172.17.3.1 any
!
ip sla 100
 icmp-echo 172.18.0.6 source-interface GigabitEthernet0/2
 threshold 500
 timeout 1000
 frequency 3
ip sla schedule 100 life forever start-time now
ip sla 101
 icmp-echo 172.18.0.6 source-interface GigabitEthernet0/3
 threshold 500
 timeout 1000
 frequency 3
ip sla schedule 101 life forever start-time now
!
route-map ISP_A permit 10
 match interface GigabitEthernet0/2
!
route-map ISP_B permit 10
 match interface GigabitEthernet0/3
!
route-map LocalPolicy permit 10
 match ip address PingISP_A
 set ip next-hop 172.16.1.2
!
route-map LocalPolicy permit 20
 match ip address PingISP_B
 set ip next-hop 172.17.3.2

Surviving Multiple Failures

The above solution survives the failure of:

• A single uplink (one of the IP SLA measurements fails)
• The central site (both IP SLA measurements fail, but we have backup floating static routes)
• The physical link between GW and PE-A (the interface on GW goes down, and even the floating static route is removed)

It does not, however, survive the failure of the central site and the path between GW and PE-A. If you want to make your solution even more reliable¹, you could combine IP SLA checks of central site reachability with IP SLA checks of next-hop reachability and use the following hierarchy of default routes:

• Route to PE-A based on reachability of the central site
• Route to PE-B based on reachability of the central site
• Route to PE-A based on next-hop reachability (using IP SLA or BFD)
• Route to PE-B based on next-hop reachability (using IP SLA or BFD)
• Floating static route to PE-A based on interface state
• Floating static route to PE-B as the absolute last resort

Revision History

2025-03-31

• Recreated the router configurations and printouts with IOSv release 15.6(1)T.
• Added the surviving multiple failures scenario at the end of this section.

The static default route configured on GW-A and GW-B has to be propagated between them to ensure that both routers have the same view of the Internet connectivity. The easiest way to implement this requirement is to redistribute the static default route into a dynamic routing protocol configured between the two routers, as shown in the next listing:

router ospf 1
 redistribute static metric 10
 default-information originate
!
interface Ethernet0/1
 ip ospf 1 area 0

If no workstations are attached to the LAN between GW-A and GW-B, we’re finished; all routers attached to that LAN will get the default route pointing to the currently active gateway router through a dynamic routing protocol (Figure 4).

Intra-site routing with workstations attached to the same LAN as GW-A and GW-B is a bit more complex. You can usually configure only a single default gateway on the workstations, so you have to provide a dynamic switchover of the default gateway with a first-hop redundancy protocol (FHRP), for example, Virtual Router Redundancy Protocol (VRRP) or Hot Standby Router Protocol (HSRP). The configuration is straightforward since the track object that you can use to adjust the router’s HRSP priority based on the state of the upstream link has already been configured (see the following two listings; the only difference is the default HSRP priority, which is higher on GW A).

interface Ethernet0/1
 ip address 192.168.0.3 255.255.255.0
 standby 1 priority 100
 standby 1 ip 192.168.0.1
 standby 1 preempt
 standby 1 track 17 decrement 20

interface Ethernet0/1
 ip address 192.168.0.4 255.255.255.0
 standby 1 priority 90
 standby 1 ip 192.168.0.1
 standby 1 preempt

The gateway routers’ configuration follows the principles explained in the Small Site Multihoming article. IP addressing and NAT are configured on both gateway routers, as shown in the following listing (only the GW-A configuration is included in most examples; the complete router configurations are on GitHub).

ip dhcp pool LAN
   network 192.168.0.0 255.255.255.0
   default-router 192.168.0.1
exit
!
ip dhcp excluded-address 192.168.0.1 192.168.0.10
ip dhcp excluded-address 192.168.0.128 192.168.0.255
!
interface Ethernet0/1
 description gw_a -> [h1,h2,gw_b]
 ip address 192.168.0.1 255.255.255.0
 ip nat enable
!
interface Ethernet0/2
 description gw_a -> pe_a
 ip address 172.16.1.1 255.255.255.252
 ip nat enable
!
ip access-list standard Site
 10 permit 192.168.0.0 0.0.0.255
!
route-map Internet_Exit permit 10
 match ip address Site
!
ip nat inside source route-map Internet_Exit interface Ethernet0/2 overload

To implement reliable static routes on both gateway routers, you have to configure:

• An IP SLA object to track end-to-end connectivity to an IP address that is “far enough” (at least within the ISP network’s core; tracking an upstream ISP server is even better).
• A track object that monitors the state of the IP SLA object.
• A local routing policy ensuring the IP SLA measurements always use the Internet interface (otherwise, a gateway router with a failed upstream link might use the default path the other gateway router provided for its SLA measurements).
• A static default route based on the state of the track object.

The relevant parts of GW-A configuration are included in the following listing (the detailed description of the configuration and monitoring commands related to reliable static routing is available in the Small Site Multihoming article).

ip access-list extended Ping_probe
 10 permit icmp host 172.16.1.1 host 172.29.0.1
!
route-map LocalPolicy permit 10
 match ip address Ping_probe
 set ip next-hop 172.16.1.2
 set interface Ethernet0/2
!
ip local policy route-map LocalPolicy
!
ip sla 15
 icmp-echo 172.29.0.1 source-interface Ethernet0/2
  threshold 100
  timeout 200
  frequency 1
ip sla schedule 15 life forever start-time now
!
track 17 ip sla 15 state
 delay down 5 up 20
!
ip route 0.0.0.0 0.0.0.0 172.16.1.2 name ISP_A track 17

The setup on GW-B is much more straightforward, as we’re using it just as a backup router. It has a floating static default route; if Internet connectivity on GW-A is operational, the default route received through the routing protocol should override the static default route.

ip route 0.0.0.0 0.0.0.0 172.17.3.2 name ISP_B 250

Connecting a small site to multiple service providers can be extremely easy – you get two upstream links and two provider-assigned (PA) IPv4 addresses (static or dynamically assigned). Since each ISP will give you only a single IPv4 address, you have to use private IPv4 addresses on the LAN side of the router (Figure 1).

Most ISPs are unwilling to run a dynamic routing protocol with small sites, so you must configure static default routing on your router. Since you would almost always prefer one provider over the other, you would create a primary and a backup default route.

The router on the remote site would also have to perform two independent NAT translations, one for packets sent to ISP A (where local addresses get translated to the IP address assigned by ISP A) and another for packets sent to ISP B (Figure 3).

One of the significant issues in multi-homed site design is the proper handling of the return traffic. It’s not uncommon to experience performance problems if the outbound and return traffic flow over different links (also known as asymmetrical routing), while IP multicast and stateful packet inspection (part of the Cisco IOS firewall feature set) almost always break under these conditions. Fortunately, asymmetrical routing is never a problem in a dual NAT design (see the above diagram), as the source address of the outbound packet indicates the link that has been used to send it: