Generate SNMP trap on high CPU load

Gernot Nusshall has asked an interesting question:

How could I configure the EEM to send an SNMP trap when the cpu load (interval=30sec) is higher than 30%?
My first solution was to enable resource policy traps with the snmp-server enable traps resource-policy, but this feature was introduced in 12.4(15)T and I am not sure everyone is willing to run the latest-and-greatest IOS code. Furthermore, it looks like the traps are sent only for resource policies defined through the ERM MIB; I was not able to generate a trap from a manually configured resource policy. Obviously it was time for another EEM applet.The EEM version 2.0 (available in 12.2S, 12.3T and 12.4) includes the action snmp-trap command, which can generate a trap from an EEM applet. To generate CPU utilization traps, configure the desired resource policy and an EEM applet that is triggered on the ERM policy event. The simplest EEM applet would just report a change in ERM policy …
event manager applet ReportHighCPU
 event resource policy "HighGlobalCPU"
 action 1.0 snmp-trap strdata "High CPU"

… but as the applet would be run on rising and falling events, it would make sense to include a few _resource_* environment variables in the SNMP trap data. Last but not least, don't forget to enable EEM traps with the snmp-server enable traps event-manager configuration command.

This article is part of You've asked for it series.

4 comments:

  1. The following (if supported by your IOS) should accomplish the same thing, without using EEM:

    process cpu threshold type total rising 30 interval 10 falling 20 interval 10
    snmp-server enable traps cpu threshold

    ReplyDelete
  2. For granularity you can use cisco SNMP mibs for cpu utilization. Below is a sample I used to disable wccp when it caused a CPU surge:

    event manager applet disable_wccp
    ! --- 1.3.6.1.4.1.9.9.109.1.1.1.1.4.1 gives the cpu utilization using 1 min interval
    ! --- trigger event when cpu utilization hits 75% and starts monitoring (re-arms) again when it drops below 40%
    ! --- SNMP polling interval is set to 60 secs
    event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.4.1 get-type exact entry-op ge entry-val 75 exit-op le exit-val 40 poll-interval 60
    ! --- logs a message
    action 101 syslog priority notifications msg "Disabling WCCP due to prolonged CPU spikes."
    ! --- disable wccp globally, this is static - you have to know the wccp svc id
    action 102 cli command "enable"
    action 103 cli command "config t"
    action 104 cli command "no ip wccp 91"
    action 105 cli command "end"

    In case you want to wrap things up with an event notification, append the following to the above disable_wccp applet:

    action 106 cli command "enable"
    action 107 cli command "terminal length 0"
    action 108 cli command "show process cpu sorted"
    action 109 mail server "192.168.1.254" to "engineer@xyz.com" from "devtest@cust_a.com" subject "Prolonged CPU Spikes" body "$_cli_result"


    And here's the reverse applet to re-enable wccp when the cpu utilization is back to normal:


    event manager applet enable_wccp
    ! --- 1.3.6.1.4.1.9.9.109.1.1.1.1.4.1 gives the cpu utilization using 1 min interval
    ! --- trigger event when cpu utilization is 40% and starts monitoring (re-arms) again when it rises to 75%
    ! --- SNMP polling interval is set to 60 secs
    event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.4.1 get-type exact entry-op le entry-val 40 exit-op ge exit-val 75 poll-interval 60
    ! --- logs a message
    action 101 syslog priority notifications msg "Enabling back WCCP as CPU surges no longer exist."
    ! --- re-enable wccp globally, this is static - you have to know the wccp svc id and redirect-list
    action 102 cli command "enable"
    action 103 cli command "config t"
    action 104 cli command "ip wccp 91 redirect-list 101"
    action 105 cli command "end"

    ReplyDelete
  3. Is it possible to see the output on the console for the cli command mentioned in the like in "action 1.0 cli command ping...". Is it possible to see the response of the ping retun ont he console?

    ReplyDelete
  4. Hi,

    Is it possible to send the first 10 lines of show process cpu with regexp.when using EEM.

    ReplyDelete

You don't have to log in to post a comment, but please do provide your real name/URL. Anonymous comments might get deleted.

Ivan Pepelnjak, CCIE#1354, is the chief technology advisor for NIL Data Communications. He has been designing and implementing large-scale data communications networks as well as teaching and writing books about advanced technologies since 1990. See his full profile, contact him or follow @ioshints on Twitter.