Category: EEM

When writing EEM applets or policies that act on CLI commands, keep in mind that the pattern matching is not context sensitive. For example, if you want to disable the reload command and use the EEM applet …

event manager applet NoReload
 event cli pattern "reload" sync no skip yes

… you cannot enter the action x.y reload configuration command any more (or any other command that includes the string reload).

To distinguish the reload command from other appearances of the same string, use the ^reload pattern (reload occuring at the beginning of the line).

Trivia: this actually occured to me when I was testing the setup described in the December IP Corner article. Sometimes we have to learn the hard way :)

The action cli commands used in EEM applets as well as the cli* Tcl functions used in EEM Tcl policies open a virtual Telnet session to a VTY line to execute the CLI commands. The first command you have to execute in the EEM applet is thus the enable command to ensure the next commands will be executed with privilege level 15.

You don't have to specify the enable password.

Likewise, if you want to configure the router, the next command to execute is the configure terminal command, followed by the configuration commands.

In response to my post about combining Tcl shell with EEM to get around the “no prompts” limitation of EEM action cli command, Xavier proposed using the undocumented pattern option of the action cli command, which changes the string the EEM script is expecting to indicate that the current command has been executed.

By default, the EEM action cli command waits until it receives exec-level prompt from the VTY (Router> or Router#), resulting in an endless wait and aborted EEM applet in IOS release 12.4(15)T (earlier releases would hang a VTY line forever) if a CLI command returns an additional prompt. With the pattern option, you can change the expected reply to whatever prompt the CLI command is outputting.

When I’ve been describing the limitations of kron, someone quickly asked an interesting question:

As I cannot insert extra input keystrokes with EEM applet, can I run a Tcl script from it with the action sequence cli command “tclsh script” command and use the typeahead function call to get around the limitation?”

The only answer I could give at that time was “maybe” … and obviously it was time for a more thorough test. The short result is: YES, you can do it (at least in IOS release 12.4(15)T1).

When two groups within Cisco needed time-based command execution in Cisco IOS, they (in a typical big-corporation fashion) decided to implement the same wheel from two different sets of spokes and rims. One group built the Embedded Event Manager with its event timer cron command (introduced in 12.2(25)S and 12.3(14)T), the other group created the more limited kron command set (introduced in 12.3(1)).

Someone has asked me a while ago whether it's possible to retain variable values between invocations of an EEM policy. Since a new copy of Tcl interpreter is started for each event, global variables obviously won't work; they are lost as soon as the Tcl policy is finished. A potential solution is to modify the router's configuration and save the values you wish to preserve in event manager environment, but that's a time-consuming process that interferes with whatever router configuration management process you have.

The real solution is based on the appl_setinfo and appl_reqinfo calls. They work, but like many other Tcl-related IOS features they are … well … weird.

William Chu sent me a working configuration he uses to measure jitter with the IP SLA tool and react to excessive jitter on the primary link. First you have to create the jitter probe with the IP SLA commands:

ip sla monitor 3000
 type jitter →
   dest-ipaddr 199.11.18.168 dest-port 12333 →
   source-ipaddr 199.11.18.169 codec g729a →
   codec-numpackets 100
 tos 184
 frequency 10

Note: The continuation character (→) indicates that the configuration command spans multiple lines

Next you have to define the IP SLA reaction to excessive jitter. William configured his router to react when the jitter exceeds 300 milliseconds and returns back to normal when the jitter falls below 290 milliseconds (some hysteresis is always a good thing).

ip sla monitor reaction-configuration 3000 →
  react MOS threshold-value 300 290 →
  threshold-type consecutive →
  action-type trapOnly

As the last step in the SLA configuration, you have to start the probe:

ip sla monitor schedule 3000 →
  life forever start-time now

After the SLA probe and out-of-bounds reaction have been configured, the router will generate syslog messages whenever the jitter gets above the threshold as well as when it falls below the second threshold. You can then use the EEM applets to act on the syslog messages:

event manager applet MOS-Below
 event syslog occurs 1 period 120 →
   pattern "Threshold below for MOS"
 ... actions ...
!
event manager applet MOS-Above
 event syslog occurs 1 period 120 →
   pattern "Threshold exceeded for MOS"
 ... actions ...

John S. Pumphrey recently asked an interesting question: “Can the router send an e-mail when an interface goes down?” The enterprisey solution is obvious: deploy a high-end EMS to collect SNMP traps and use its API to write a custom module that would use a MQ interface to alert the operator. Fortunately, Event Manager applets in Cisco IOS provide action mail command (available in 12.3(14)T and 12.4) that can send an e-mail to a SMTP server straight from the router.

There are two ways you can detect that an interface went down with EEM: either you track the interface status with a track object and start an EEM applet when the track object changes state or you catch the syslog messages reporting that the interface line protocol changed state to down. The second approach is obviously more generic, as a single applet can act on multiple interfaces.

event manager applet MailOnIfDown
 event syslog occurs 1 →
    pattern "LINEPROTO-5-UPDOWN.*to down" →
    period 1

The action mail command specifies the mail server's address (use a hostname and DNS lookup or ip host configuration command to make the EEM applet more generic), from and to address, message subject and body. In each of these fields, you can use EEM environment variables that you can define with the event manager environment configuration command. Each EEM event also defines a few environment variables that you can use (see the table of EEM system-defined variables on CCO). For example, you can define the e-mail recipient in the router's configuration and use the _syslog_msg variable to include the syslog message in the e-mail body:

event manager environment _ifDown_rcpt [email protected]
!
event manager applet MailOnIfDown
 event syslog occurs 1 →
    pattern "LINEPROTO-5-UPDOWN.*to down" →
    period 1
 action 1.0 mail server "mail-gw" →
    to "$_ifDown_rcpt" from "[email protected]" →
    subject "Interface down on R1" →
    body "$_syslog_msg"

You can make the applet even more generic with the help of action info type routername command, which stores the current router's name into the $_info_routername environment variable:

event manager environment _ifDown_rcpt [email protected]
!
event manager applet MailOnIfDown
 event syslog occurs 1 →
    pattern "LINEPROTO-5-UPDOWN.*to down" →
    period 1
 action 1.0 info type routername
 action 2.0 mail server "mail-gw" →
    to "$_ifDown_rcpt

" from "$_info_routername@lab.com" →
    subject "Interface down on $_info_routername" →
    body "$_syslog_msg"

Note: This article is part of You've asked for it series.

If you use EEM applets with action syslog commands, the messages generated by those commands will be displayed in the same format as the debugging messages, regardless of their severity. This might be a problem if you decide to use different timestamps for debugging and regular syslog messages (or disable debugging timestamps completely).

It's a bit embarrassing: I've posted a hint on how to execute commands after a router reload and Aamer figured out something I've wanted to have for years: the ability to automatically start debugging after the router reload.

The IOS documentation claims that the quiet mode the router enters after a series of login failures blocks all telnet (or ssh) sessions as well as HTTP requests. Unfortunately the latter is wrong; you can execute any HTTP request on the router during the quiet mode.

If you want to block HTTP requests during the quiet mode, you can use EEM applets to change the HTTP server configuration when the quiet mode is started and completed.
First you need to configure a standard numbered IP access list that will be used to block HTTP requests during the quiet mode (the ip http access-class command accepts only numbered ACLs), for example:

access-list 95 deny any log

Then you define two EEM applets: one that triggers when the router enters the quiet mode (matching the SEC_LOGIN-1-QUIET_MODE_ON syslog message) and another that runs when the quiet mode is finished (triggered with the SEC_LOGIN-5-QUIET_MODE_OFF). Both applets modify the router configuration, changing the access-list used in ip http access-class configuration command.

event manager applet EnterQuietMode
 event syslog occurs 1 pattern "SEC_LOGIN-1-QUIET_MODE_ON" period 1
 action 1.0 cli command "configure terminal" 
 action 1.1 cli command "ip http access-class 95" 
 action 2.0 syslog msg "Entered Quiet mode on HTTP server"
!
event manager applet ExitQuietMode
 event syslog occurs 1 pattern "SEC_LOGIN-5-QUIET_MODE_OFF" period 1
 action 1.0 cli command "configure terminal"
 action 1.1 cli command "ip http access-class 70"
 action 2.0 syslog msg "Exiting Quiet mode on HTTP server"

Testing Embedded Event Manager (EEM) Tcl policies is a convoluted process:

• Source file is usually edited on a general-purpose workstation.
• The file has to be downloaded to router's local storage (EEM does not register non-local policies).
• The new version of the EEM policy has to be registered with EEM with event manager policy configuration command
• After all these steps, the new policy can be tested.

While you can use EEM applet to automate this process, slightly more flexible approach (you can specify the policy name to be replaced) can be implemented with Tcl script:

set policy [lindex $argv 0]
set source "tftp://10.0.0.10/tcl/" # replace with your host and directory
set destination "nvram:" # replace with local storage device
if {[string equal $policy ""]} {
return -code error "expected policy name"
}
puts "replacing policy: $policy"
ios_config "file prompt quiet"
ios_config "no event manager policy $policy" ""
exec "copy $source$policy $destination$policy"
ios_config "event manager policy $policy"
ios_config "no file prompt quiet"

To use the script, follow these steps:

• Save the script in a .tcl file (for example, changePolicy.tcl)
• Change the script parameters (remote host and local storage)
• Save the .tcl file to your router's local storage (you can also run it from a remote server)
• Configure a command alias, for example alias exec eem tclsh flash:changePolicy.tcl testPolicy.tcl

Now you can replace the target EEM Tcl policy with a simple eem command.

Alternatively, if you define alias exec eem tclsh flash:changePolicy.tcl, you can specify policy name as an argument to the eem command, for example eem testPolicy.tcl.

If you want to execute IOS show commands periodically (for example, to monitor router status or take snapshots of routing tables), you can combine new output redirection features introduced in IOS release 12.2T in an Embedded Event Manager (EEM) applet. For example, to store the brief interface status into a file on an FTP server, use the following EEM applet:

event manager applet SaveInterfaceStatus
 event timer watchdog name SaveIfStat time 60
 action 1.0 cli command "show ip interface brief | redirect ftp://username@password:host/path"
 action 2.0 syslog msg "Interface status saved"

Notes:

• The timer watchdog EEM event defines a recurring event triggered every X seconds.
• Output of a show command can be redirected only to a TFTP or FTP server, redirection to a web (HTTP) server does not work yet.
• The syslog action is configured for debugging purposes only and can be removed in production environment.
• More complex functionality (for example, sending show command output in an email) can be implemented with help of Tcl EEM policies

One of my readers asked an interesting question: „why do the commands executed within a EEM Tcl policy fail with Command authorization fails message?“ The short answer is simple: If you use AAA command authorization (which you can only do if you're using a TACACS+ server), you have to specify the username under which the EEM will execute its CLI commands with the event manager session cli username user configuration command.

Sometimes, parts of router configuration get lost during the reload process: although the configuration commands are saved in NVRAM, they are not processed after the reload and thus do not appear in the running configuration. Re-entering these commands manually solves the problem ... but it's obviously not a reliable solution.

Embedded Event Manager (EEM) solves this issue as well. You just configure an applet that triggers on syslog message SYS-5-RESTART and reapplies the necessary configuration commands.