Blog Posts in March 2008
Tcl-based IOS backdoor
Andy Davis from the Information Risk Management has written an interesting "application note": how to create a backdoor to Cisco IOS using Tclsh (I've tested it and it works quite nicely). His backdoor implementation relies on a bug in Cisco IOS that allows a "hung" (or never-ending) Tclsh script to continue executing even after the user session has been disconnected (the only means of stopping such a script is with the clear line command). That bug has been fixed in recent IOS versions (I've tested that as well, the Tclsh script is killed as soon as the Telnet session is disconnected in IOS 12.4(15)T), but you can still use the same technique (although it might be a bit less convenient) if you convert the Tclsh script into an EEM policy and trigger it periodically with a timer event.
Track the DHCP Default Route
Cisco has published a series of documents describing how you can connect a SOHO site to two ISPs.
Their configuration also includes a nice trick: the ip dhcp client route track number command is a convenient replacement for a static default route with the track option if one of the upstream interfaces uses DHCP and the router generates the default route based on DHCP replies.
NAT activates NBAR
A few days ago I had an “interesting” experience on a router that was running low on memory: when I enabled NAT, it immediately ran out of memory although it had over 4 MB free memory before that (and since I was doing the tests in a lab, I wasn't worried about that … in a production network, 4 MB of free memory is something to worry about).
It took me a while to figure out what was going on: the moment you enable NAT in IOS release 12.4, it activates Network Based Application Recognition (NBAR) even when CEF is disabled (and supposedly NBAR requires CEF to run).
Detect routers operating in process-switching mode
Sometimes the CPU utilization on a router would raise unexpectedly due to incoming packets being process switched. A very common scenario is a GRE tail-end router that has to reassemble IP fragments (usually generated due to incorrect MTU size on the GRE head-end or due to IPSec+GRE combination) or a router under Denial-of-Service attack. To detect these conditions, you can define Embedded Resource Manager (ERM) policy that raises an alert when the CPU utilization of the IP Input process exceeds predefined limits.
Predefine your own Tcl functions
If you want to have your own Tcl functions available when you start tclsh, you could use the scripting tcl init file configuration command that I've briefly mentioned in one of the previous posts. This command specifies a source file that is executed every time you start Tcl shell. The source file can contain function definitions, package declarations or any other Tcl code.
If you need to, you can specify multiple initialization files.
Local-AS Has to Be Matched by Incoming Filter-List
In a previous post I've described how you can use neighbor local-as feature to fix AS-number mismatch between adjacent autonomous systems. However, without additional options, the local-as is inserted in the AS-path of incoming BGP updates before any inbound filters. Your inbound filters thus have to match the local-as as well.
When “copy” actually means “merge”
Marcus Jensen asked me a very interesting question:
I want to send 3 lines of configuration to a remote router, but I know the first line will kill my connection. Can I save these 3 lines of code to a text file, and then issue a Tcl command to add those to the running config?
The solution is much simpler and does not have to involve Tcl at all. The copy something system:running-config command merges the configuration commands in the source file with the current running configuration.
SNTP will not work if you've configured NTP
If you're running NTP and SNTP on the same router, SNTP will never synchronize with a configured SNTP server, as the NTP process captures the reply packet before SNTP has the chance to react.
SNTP multicast/broadcast client mode works in combination with NTP
NTP process could be running even if your running configuration has no NTP-related commands. It starts automatically whenever you enter NTP-related configuration (ntp logging configuration command is enough) and is not stopped when the last NTP-related configuration command is removed. You have to reload the router to kill it.
Use EEM to respond to ERM events
In a previous post, I've described how you can detect high CPU load with the Embedded Resource Manager (ERM). If you want to respond to these events, you could use the syslog event detector within EEM, but it's more reliable to use the new event resource detector available in EEM version 2.2 (introduced in IOS release 12.4(2)T). The resource detector is best used in Tcl policy; if you use it in EEM applet, the same applet is triggered every time a resource policy threshold (minor/major/critical, rising or falling) is crossed. Within the EEM applet it's almost impossible to detect which threshold was crossed.
How Do I Detect Router Restarts?
Mike Nipp has wondered which syslog message to use to reliably detect router reload under all circumstances:
The problem I had with the SYS-5-RESTART message is I don't think you will get one if the power is suddenly pulled from the router. It does do a SNMP-5-COLDSTART and SYS-6-BOOTTIME on boot up.
I did an actual power-cycle test of a router and the SYS-5-RESTART message is reliably generated at every startup, be it from the power cycle or the reload command (I was not able to provoke an on-demand crash ;).
The Mysteries of the “Internet” BGP Community
Cisco documentation has always claimed there were four well-known communities (the Internet community being one of them), while the RFC 1997 lists three well-known values. Unfortunately, many people blindly copy the IOS documentation without asking themselves “what the heck is the Internet community”.
Detect CPU spikes with Embedded Resource Manager
David Winter wanted to detect high-CPU spikes and act on them. The first part (high CPU utilization) could be done with SNMP, but since IOS release 12.3(14)T, the right tool for the job is the Embedded Resource Manager (ERM).
The ERM syntax is a bit baroque (and not well documented), so let's work through the example: this is the configuration you need to detect high overall CPU utilization on the main CPU in the box:
Display the names of the configured route-maps
I’m probably getting old … I keep forgetting the exact names (and capitalization) of route-maps I’ve configured on the router. The show route-maps command is way too verbose when I’m simply looking for the exact name of the route-map I want to use, so I wrote a Tcl script that displays the names of the route-maps configured on the router. If you add the -d switch, it also displays their descriptions (specifically, the first description configured in the route-map).
Copy file to an FTP server with EEM applet
cpmf14 has left an interesting comment documenting how to perform a periodic back up of a file in router's flash to an FTP server:
event manager applet backup-crl
event timer watchdog time 86400 maxrun 4294967295
action 1.0 cli command "enable"
action 2.0 cli command "copy flash:/iosca.crl ftp://username:[email protected]/" pattern "a.b.c.d"
action 3.0 cli command "a.b.c.d" pattern "iosca.crl"
action 4.0 cli command "iosca.crl"
action 5.0 syslog msg "FTP backup successful"
Router Fragmentation Is Gone from IPv6
In response to my Never-Ending Story of IP Fragmentation, Stojanco Cavdarov made an interesting observation: routers are not allowed to fragment IPv6 packets, they have to respond back with ICMP unreachable (effectively, routers behave as if IPv6 packets would have an implicit don't fragment bit).
To make life easier for non-TCP IPv6 applications (TCP is supposed to use Path MTU Discovery), the minimum IPv6 packet size that has to be supported on all links was increased to 1280 bytes (which, incidentally, fits very nicely into GRE+IPSec envelope transported across links with 1500-byte MTU).
WAN IP addresses and subnet masks
Whisper asked an interesting question …
“What I would like to know is, on my PPP negotiated ADSL connection, how the ISP assigns me a /32 ip address.”
… which prompted me to test various WAN encapsulations and address assignment rules. Here are the results:
Phase 2: Upload text files through a Telnet session
In a previous post, I've described how you can use Tcl shell to upload text content into the router's flash if the router has no connectivity to a suitable file server (or you don't have FTP or TFTP server handy).
The trick works flawlessly, but typing the same obscure Tcl commands gets tedious after a while, so the first time I had to use this solution to develop a Tcl script, I quickly wrote another script that takes a file name as a parameter and hides all the other murky details.
Debugging time-based configuration
Debugging time-based configurations could be a nightmare, as you have to switch router's time back and forth trying to debug your configuration and wait for the desired event to occur. When I was debugging my EEM-based solution to time-based BGP policy routing, I simply defined two aliases that would set the clock to 30 seconds before the event I wanted to test:
alias exec 859 clock set 08:59:30
alias exec 900 clock set 09:00:30
Obviously, these tests are best done in a lab setup … and you have to turn off NTP or any other form of time synchronization.