Dampen Other Instabilities

Articles » Increase the Stability of your Network » Dampen Other Instabilities

The external BGP routes and physical interface flaps are the only network instabilities that can be dampened using standard Cisco IOS features. To dampen any other instabilities in your network (for example, reloading routers, lost neighbors on LAN interfaces, or flapping subinterfaces), you must write your own Embedded Event Manager (EEM) applets or Tcl scripts. The EEM applets are usually applied as a temporary fix, as you have to use a different event definition for each interface or OSPF neighbor. Thus, you must configure a separate applet for each interface or neighbor router. Using Tcl scripts registered as EEM policies gives you greater flexibility (you could even implement exponential decay dampening using Tcl), but it is well beyond the scope of this article.

The best method to detect network instability is to monitor the syslog messages generated by the router (for example, the OSPF-5-ADJCHG messages) and trigger an action if the same message is generated too often in a specified time frame. For instance, you could trigger an EEM applet if the OSPF neighbor on interface Serial0/1/0 is lost more than three times in a minute:

Detect repeated OSPF neighbor loss on a serial interface
event manager applet OSPF_Down_Serial0/1/0
 event syslog occurs 3 pattern "OSPF-5-ADJCHG.*Serial0/1/0.*to DOWN" period 60

Once the network instability is detected, you might want to take proactive steps to reduce its impact. For example, if an OSPF adjacency is constantly failing on an interface, you could make that interface passive:

Making an OSPF interface passive with an EEM applet
event manager applet OSPF_Down_Serial0/1/0
 event syslog occurs 3 pattern "OSPF-5-ADJCHG.*Serial0/1/0.*to DOWN" period 60
 action 1.0 cli command "enable"
 action 2.0 cli command "configure terminal"
 action 2.1 cli command "router ospf 1"
 action 2.2 cli command "passive-interface serial 0/1/0"
 action 3.0 syslog msg "OSPF disabled on Serial0/1/0"
When taking such a radical step, you might want to notify the network administrator via email (potentially forwarded to a mobile phone). The action mail EEM configuration command allows you to send emails from an EEM applet.

You could decide that the router will just disable a discovered network instability and inform the network administrator, or you could trigger a reverse configuration change after some time. To do that, you have to define another EEM applet with reverse effects and trigger that applet with a countdown timer:

Re-enable OSPF on a serial interface
event manager applet OSPF_Enable_Serial0/1/0
 event none
 action 1.0 cli command "enable"
 action 2.0 cli command "configure terminal"
 action 2.1 cli command "router ospf 1"
 action 2.2 cli command "no passive-interface serial 0/1/0"
 action 3.0 syslog msg "OSPF enabled on Serial0/1/0"
Trigger the complementary EEM applet in 60 seconds
event manager applet OSPF_Down_Serial0/1/0
 action 4.0 cli command "event manager applet OSPF_Enable_Serial0/1/0"
 action 4.1 cli command "event timer countdown name OSPF_Enable_Serial0/1/0 time 60"

The following printout contains the complete EEM applet configuration:

Complete EEM configuration to detect OSPF instabilities on Serial0/1/0
event manager applet OSPF_Down_Serial0/1/0
 event syslog occurs 3 pattern "OSPF-5-ADJCHG.*Serial0/1/0.*to DOWN" period 60
 action 1.0 cli command "enable"
 action 2.0 cli command "configure terminal"
 action 2.1 cli command "router ospf 1"
 action 2.2 cli command "passive-interface serial 0/1/0"
 action 3.0 syslog msg "OSPF disabled on Serial0/1/0"
 action 4.0 cli command "event manager applet OSPF_Enable_Serial0/1/0"
 action 4.1 cli command "event timer countdown name OSPF_Enable_Serial0/1/0 time 60"
event manager applet OSPF_Enable_Serial0/1/0
 event timer countdown name OSPF_Enable_Serial0/1/0 time 60
 action 1.0 cli command "enable"
 action 2.0 cli command "configure terminal"
 action 2.1 cli command "router ospf 1"
 action 2.2 cli command "no passive-interface serial 0/1/0"
 action 3.0 syslog msg "OSPF enabled on Serial0/1/0"

A sample scenario where three interface flaps have been generated far enough apart that the IP Event Dampening did not detect them (but EEM detected OSPF instability) might generate logging messages similar to the following printout:

EEM applet disables and reenables OSPF on a serial interface
Aug 14 11:02:15: %LINK-3-UPDOWN: Interface Serial0/1/0, changed state to down
Aug 14 11:02:15: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.0.12 on Serial0/1/0 from FULL to DOWN, Neighbor Down: Interface down or detached
Aug 14 11:02:21: %LINK-3-UPDOWN: Interface Serial0/1/0, changed state to up
Aug 14 11:02:27: %LINK-3-UPDOWN: Interface Serial0/1/0, changed state to down
Aug 14 11:02:27: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.0.12 on Serial0/1/0 from INIT to DOWN, Neighbor Down: Interface down or detached
Aug 14 11:02:33: %LINK-3-UPDOWN: Interface Serial0/1/0, changed state to up
Aug 14 11:02:34: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/1/0, changed state to up
Aug 14 11:02:40: %LINK-3-UPDOWN: Interface Serial0/1/0, changed state to down
Aug 14 11:02:40: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.0.12 on Serial0/1/0 from INIT to DOWN, Neighbor Down: Interface down or detached
Aug 14 11:02:40: %HA_EM-6-LOG: OSPF_Down_Serial0/1/0: OSPF disabled on Serial0/1/0
Aug 14 11:02:40: %SYS-5-CONFIG_I: Configured from console by vty0
Aug 14 11:02:46: %LINK-3-UPDOWN: Interface Serial0/1/0, changed state to up
Aug 14 11:03:40: %HA_EM-6-LOG: OSPF_Enable_Serial0/1/0: OSPF enabled on Serial0/1/0
Aug 14 11:03:40: %SYS-5-CONFIG_I: Configured from console by vty0
Aug 14 11:03:46: %OSPF-5-ADJCHG: Process 1, Nbr 172.16.0.12 on Serial0/1/0 from LOADING to FULL, Loading Done

Continue

Next: Summary

Sidebar