My recent post about a CCNP bootcamp program I was involved with generated interesting comments, most of them focusing on the question: “Does it make sense to attend a bootcamp?”

The answer depends on how you got to the stage where you want to (or need to) attain the CCxP certification. Before going into discussions on “experience” versus “knowledge retention” (potentially “aided” by brain dumps), please read The Top 10 Problems with IT Certification in 2008 article published by InformIT. My potential disagreements with this article are so minor that I will not even try to document them.

OK, now that we're on the same page, let's analyze why someone would want to pursue CCxP certification:

• To increase the salary or have better job options (as HR departments ask for people with specific set of certifications). From what I hear, this reason is more viable in US than the rest of the world (in most of Europe we can still test the technical skills of the candidates in any way we want without running the risk of being sued). Bootcamps might not be the best option for these candidates, as they tend to be priced similarly to the regular classes. Reading books or e-learning material (not to mention certain not-so-very-legitimate activities) will get you through the exams as long as they don't have the hands-on part ... and of course you'll end up having certification with zero experience;
• To learn something new and valuable resulting in a formal recognition of the effort. Don't even think about attending the bootcamps. If you're learning completely new concepts, go through the regular courses (or use e-learning combined with hands-on lab exercises). Highly intensive format of the bootcamps (after all, we're trying to squeeze almost two weeks worth of material into a single week) will fly way over your head.
• To formalize your experience ... either because you want to or because your employer needs certified head count (very common with Cisco partners trying to get better discount based on their partner status). In this case, a condensed bootcamp is usually the best option. For example, we had very successful bootcamp program a few years ago running back-to-back with the exams ... and, mind you, we used no cheating or brain dumps, the fact that the students took the exam right after the course obviously helped.

Last but definitely not least, it's worth mentioning that not all five-day courses have five days worth of content. In these cases, condensing them into bootcamps makes even more sense.

The information on the IOS XE software used by the recently launched ASR 1000 router is pretty scarce (there is still no link to the documentation available on CCO), but obviously some backdoor links already exist, as I was able to find some IOS XE-related documents with Google. One of the most amazing features I've found is the support for the security-oriented RFC 3514 which allows you to mark the security level of an IP packet.

The RFC 3514 requires the end host to participate in the process, but as most operating system vendors still don't have a trusted computing platform, a transparent proxy has to be implemented on the network edges to properly tag the ingress packets. ASR 1000 has the first high-speed implementation of the RFC 3514 proxy thanks to its non-deterministic parallel QuantumFlow processors.

The configuration of the RFC 3514 proxy is extremely simple: all you need to do is to configure auto-secure mark on the ingress interfaces of the ASR 1000. Once the security bit has been set, you can use the match ip security-bit 0|1 command in a class-map or a route-map on any router running IOS release 12.4(11)T or later (the command is still hidden).

Andy Davis from the Information Risk Management has written an interesting "application note": how to create a backdoor to Cisco IOS using Tclsh (I've tested it and it works quite nicely). His backdoor implementation relies on a bug in Cisco IOS that allows a "hung" (or never-ending) Tclsh script to continue executing even after the user session has been disconnected (the only means of stopping such a script is with the clear line command). That bug has been fixed in recent IOS versions (I've tested that as well, the Tclsh script is killed as soon as the Telnet session is disconnected in IOS 12.4(15)T), but you can still use the same technique (although it might be a bit less convenient) if you convert the Tclsh script into an EEM policy and trigger it periodically with a timer event.

Cisco has published a series of documents describing how you can connect a SOHO site to two ISPs.

Their configuration also includes a nice trick: the ip dhcp client route track number command is a convenient replacement for a static default route with the track option if one of the upstream interfaces uses DHCP and the router generates the default route based on DHCP replies.

A few days ago I had an “interesting” experience on a router that was running low on memory: when I enabled NAT, it immediately ran out of memory although it had over 4 MB free memory before that (and since I was doing the tests in a lab, I wasn't worried about that … in a production network, 4 MB of free memory is something to worry about).

It took me a while to figure out what was going on: the moment you enable NAT in IOS release 12.4, it activates Network Based Application Recognition (NBAR) even when CEF is disabled (and supposedly NBAR requires CEF to run).

Sometimes the CPU utilization on a router would raise unexpectedly due to incoming packets being process switched. A very common scenario is a GRE tail-end router that has to reassemble IP fragments (usually generated due to incorrect MTU size on the GRE head-end or due to IPSec+GRE combination) or a router under Denial-of-Service attack. To detect these conditions, you can define Embedded Resource Manager (ERM) policy that raises an alert when the CPU utilization of the IP Input process exceeds predefined limits.

resource policy
  policy HighProcCPU type iosprocess
   system
    cpu process
     critical rising 40 falling 25
     major rising 20 falling 10
    !
   !
  !

  user group IPInput type iosprocess
   instance "IP Input"
   policy HighProcCPU

And here are some more ERM usage guidelines:

• This time, we're monitoring a group of processes, so the policy definition is no longer global but has a type (iosprocess is the only type defined at the moment).
• As in the previous ERM example, we're monitoring CPU utilization of the main CPU (system), but this time we're interested in the process utilization.
• The policy is applied to a user group of resources of the type iosprocess (translated into English: a group of IOS processes).
• The only process in this group is the IP Input process (and the "magic keyword" is an instance of the group).

The quotes in the instance configuration command are required, as the command accepts only a single word as the process name.

If you want to have your own Tcl functions available when you start tclsh, you could use the scripting tcl init file configuration command that I've briefly mentioned in one of the previous posts. This command specifies a source file that is executed every time you start Tcl shell. The source file can contain function definitions, package declarations or any other Tcl code.

If you need to, you can specify multiple initialization files.

For example, if you'd like to implement a comfortable Tcl-based pinger (similar to the one Ethan Banks found in the Sadikhov forums, store the following Tcl code into the file flash:pinger.tcl …

proc pinger { iplist } {
  foreach ip $iplist {
    if { [regexp "(!!!)" [exec "ping $ip timeout 1" ]] } {
      puts "$ip"
    } else { puts "$ip **** failed ***" }
  }
}

… and configure scripting tcl init flash:pinger.tcl. Now you can ping a number of hosts in a single operation:

R1#tclsh
R1(tcl)#pinger { 10.0.0.1 10.0.0.2 10.0.0.3 10.0.0.4 }
10.0.0.1
10.0.0.2
10.0.0.3 **** failed ***
10.0.0.4 **** failed ***

In a previous post I've described how you can use neighbor local-as feature to fix AS-number mismatch between adjacent autonomous systems. However, without additional options, the local-as is inserted in the AS-path of incoming BGP updates before any inbound filters. Your inbound filters thus have to match the local-as as well.

Marcus Jensen asked me a very interesting question:

I want to send 3 lines of configuration to a remote router, but I know the first line will kill my connection. Can I save these 3 lines of code to a text file, and then issue a Tcl command to add those to the running config?

The solution is much simpler and does not have to involve Tcl at all. The copy something system:running-config command merges the configuration commands in the source file with the current running configuration.

You can store the configuration commands you want to execute in a local file (even in NVRAM) or you could execute them directly off a file server (using HTTP, FTP, TFTP or SCP protocol).

This article is part of You've asked for it series.

If you're running NTP and SNTP on the same router, SNTP will never synchronize with a configured SNTP server, as the NTP process captures the reply packet before SNTP has the chance to react.

SNTP multicast/broadcast client mode works in combination with NTP

NTP process could be running even if your running configuration has no NTP-related commands. It starts automatically whenever you enter NTP-related configuration (ntp logging configuration command is enough) and is not stopped when the last NTP-related configuration command is removed. You have to reload the router to kill it.

In a previous post, I've described how you can detect high CPU load with the Embedded Resource Manager (ERM). If you want to respond to these events, you could use the syslog event detector within EEM, but it's more reliable to use the new event resource detector available in EEM version 2.2 (introduced in IOS release 12.4(2)T). The resource detector is best used in Tcl policy; if you use it in EEM applet, the same applet is triggered every time a resource policy threshold (minor/major/critical, rising or falling) is crossed. Within the EEM applet it's almost impossible to detect which threshold was crossed.

However, even EEM applet could solve some immediate problems. For example, if you want to store a snapshot of processes on a TFTP server every time the global CPU load crosses a policy threshold, you could use the following applet:

event manager applet ReportHighCPU
 event resource policy "HighGlobalCPU"
 action 1.0 cli command "show process cpu sorted 5sec | redirect tftp://10.0.0.10/highCPU$_resource_time_sent.txt"

To differentiate the snapshots, I've appended the _resource_time_sent variable set by the EEM before the applet is started to the file name, guaranteeing that the snapshot files will have unique names (at least until the router reload).

As an alternative, you could send the show process output in an e-mail:

event manager environment _ifDown_rcpt [email protected]
!
event manager applet ReportHighCPU
 event resource policy "HighGlobalCPU"
 action 1.0 cli command "show process cpu sorted 5sec"
 action 1.1 info type routername
 action 2.0 mail server "mail-gw" →
    to "$_ifDown_rcpt" from "[email protected]" →
    subject "CPU @ $_resource_current_value" →
    body "$_cli_result"

This article is part of You've asked for it series.

Mike Nipp has wondered which syslog message to use to reliably detect router reload under all circumstances:

The problem I had with the SYS-5-RESTART message is I don't think you will get one if the power is suddenly pulled from the router. It does do a SNMP-5-COLDSTART and SYS-6-BOOTTIME on boot up.

I did an actual power-cycle test of a router and the SYS-5-RESTART message is reliably generated at every startup, be it from the power cycle or the reload command (I was not able to provoke an on-demand crash ;).

Cisco documentation has always claimed there were four well-known communities (the Internet community being one of them), while the RFC 1997 lists three well-known values. Unfortunately, many people blindly copy the IOS documentation without asking themselves “what the heck is the Internet community”.

David Winter wanted to detect high-CPU spikes and act on them. The first part (high CPU utilization) could be done with SNMP, but since IOS release 12.3(14)T, the right tool for the job is the Embedded Resource Manager (ERM).

The ERM syntax is a bit baroque (and not well documented), so let's work through the example: this is the configuration you need to detect high overall CPU utilization on the main CPU in the box:

resource policy
 policy HighGlobalCPU global
  system
   cpu total
    critical rising 95 falling 70 interval 10
    major rising 75 falling 50 interval 10
 !
 user global HighGlobalCPU

And here are the usage/configuration guidelines:

• The whole ERM subsystem is configured under the resource policy section;
• You always have to configure a policy and a user to which the policy applies. In our example, the user is global (as we're measuring the global CPU load);
• The policy we're defining must have the global keyword to indicate we're measuring overall utilization (otherwise you can't attach it to the global user);
• We're measuring the load on the main CPU, so we're configuring the system subsection of the policy (on distributed platforms you could specify slot name to measure utilization on a specific linecard);
• The cpu section selects CPU load measurements. You could measure interrupt load, process load or total CPU load.
• Within each resource section in the policy (in our example, total CPU load on the main system) you can define minor, major and critical thresholds (syslog messages are generated when each threshold is crossed).
• After the policy is defined, it's applied to the global user.

With the CPU load measurement policy defined, the router will generate syslog messages (SYS-4-CPURESRISING) every time the overall CPU load exceeds the specified rising thresholds. When the utilization falls below the falling threshold, the SYS-4-CPURESFALLING syslog message is generated.

This article is part of You've asked for it series.

I’m probably getting old … I keep forgetting the exact names (and capitalization) of route-maps I’ve configured on the router. The show route-maps command is way too verbose when I’m simply looking for the exact name of the route-map I want to use, so I wrote a Tcl script that displays the names of the route-maps configured on the router. If you add a -d switch, it also displays their descriptions (to be more precise, the first description configured in the route-map).