Andy Davis from the Information Risk Management has written an interesting "application note": how to create a backdoor to Cisco IOS using Tclsh (I've tested it and it works quite nicely). His backdoor implementation relies on a bug in Cisco IOS that allows a "hung" (or never-ending) Tclsh script to continue executing even after the user session has been disconnected (the only means of stopping such a script is with the clear line command). That bug has been fixed in recent IOS versions (I've tested that as well, the Tclsh script is killed as soon as the Telnet session is disconnected in IOS 12.4(15)T), but you can still use the same technique (although it might be a bit less convenient) if you convert the Tclsh script into an EEM policy and trigger it periodically with a timer event.

Cisco has recently published a series of documents describing how you can connect a SOHO site to two ISPs. According to the timestamps on the documents, it took them a few days more than a year to publish a configuration that's almost identical to the one I've described in the IP Corner article “Small-site multihoming” … which can only mean that my configuration was close to optimal :)

Their configuration also includes a nice trick: the ip dhcp client route track number command is a convenient replacement for a static default route with the track option if one of the upstream interfaces uses DHCP and the router generates the default route based on DHCP replies.

If you want study for your CCNP certification by attending live instructor-led courses, you'll have to spend four weeks in a classroom. That's "a bit" too much for some of our customers, so we've been asked to create a shorter version of the courses, resulting in a 2-week very intensive bootcamp structure.

The CCNP Bootcamp Part 1 combines the content of the first two prerequisite courses for CCNP: Building Scalable Cisco Internetworks (BSCI) and Building Cisco Multilayer Switched Networks (BCMSN). It's an intensive combination of lecture and practice covering routing protocols and LAN switching.

The CCNP Bootcamp Part 2 combines the content of the second two prerequisite courses for CCNP: Implementing Secure Converged Wide Area Networks (ISCW) and Optimizing Converged Cisco Networks (ONT). This one obviously contains lecture and practice covering secure remote access into core networks using VPN technologies (such as MPLS VPN and IPsec VPN), network security, Quality of Service (QoS) for optimized voice transport as well as wireless security.

It's amazing how many people have load-balancing-related issues. I get asked the same question over and over:

“Is it possible to have unequal-cost load balancing with OSPF?”

The answer is invariably: NO, you cannot do it with OSPF. However, you can use MPLS Traffic Engineering to establish two tunnels to the same remote OSPF router. Both tunnels will be used for all destinations reachable through the remote OSPF router (tunnel tail-end) even though OSPF selects only a single best path to it. A simple scenario is described in my IP Corner article “Perfect Load-Balancing: How Close Can You Get?”

This article is part of You've asked for it series.

A few days ago I had an “interesting” experience on a router that was running low on memory: when I enabled NAT, it immediately ran out of memory although it had over 4 MB free memory before that (and since I was doing the tests in a lab, I wasn't worried about that … in a production network, 4 MB of free memory is something to worry about).

It took me a while to figure out what was going on: the moment you enable NAT in IOS release 12.4, it activates Network Based Application Recognition (NBAR) even when CEF is disabled (and supposedly NBAR requires CEF to run).

Here's a sample test: the moment I've configured a loopback interface to be NAT inside interface (and it was the only NAT-enabled interface in the box), NBAR consumed 4.5 MB of memory:

R2(config)#int loop 0
R2(config-if)#ip nat inside
R2(config-if)#do show ip nbar resources
NBAR memory usage for tracking Stateful sessions
   System link age      : 30 secs
   Initial memory       : 4455 KBytes
   Max initial memory   : 14852 KBytes
   Memory expansion     : 112 KBytes
   Max memory expansion : 112 KBytes
   Memory in use        : 4455 KBytes
   Max memory allowed   : 29705 KBytes
   Active links         : 0
   Total links          : 39784

Sometimes the CPU utilization on a router would raise unexpectedly due to incoming packets being process switched. A very common scenario is a GRE tail-end router that has to reassemble IP fragments (usually generated due to incorrect MTU size on the GRE head-end or due to IPSec+GRE combination) or a router under Denial-of-Service attack. To detect these conditions, you can define Embedded Resource Manager (ERM) policy that raises an alert when the CPU utilization of the IP Input process exceeds predefined limits.

resource policy
  policy HighProcCPU type iosprocess
   system
    cpu process
     critical rising 40 falling 25
     major rising 20 falling 10
    !
   !
  !

  user group IPInput type iosprocess
   instance "IP Input"
   policy HighProcCPU

And here are some more ERM usage guidelines:

• This time, we're monitoring a group of processes, so the policy definition is no longer global but has a type (iosprocess is the only type defined at the moment).
• As in the previous ERM example, we're monitoring CPU utilization of the main CPU (system), but this time we're interested in the process utilization.
• The policy is applied to a user group of resources of the type iosprocess (translated into English: a group of IOS processes).
• The only process in this group is the IP Input process (and the "magic keyword" is an instance of the group).

The quotes in the instance configuration command are required, as the command accepts only a single word as the process name.

If you want to have your own Tcl functions available when you start tclsh, you could use the scripting tcl init file configuration command that I've briefly mentioned in one of the previous posts. This command specifies a source file that is executed every time you start Tcl shell. The source file can contain function definitions, package declarations or any other Tcl code.

If you need to, you can specify multiple initialization files.

For example, if you'd like to implement a comfortable Tcl-based pinger (similar to the one Ethan Banks found in the Sadikhov forums, store the following Tcl code into the file flash:pinger.tcl …

proc pinger { iplist } {
  foreach ip $iplist {
    if { [regexp "(!!!)" [exec "ping $ip timeout 1" ]] } {
      puts "$ip"
    } else { puts "$ip **** failed ***" }
  }
}

… and configure scripting tcl init flash:pinger.tcl. Now you can ping a number of hosts in a single operation:

R1#tclsh
R1(tcl)#pinger { 10.0.0.1 10.0.0.2 10.0.0.3 10.0.0.4 }
10.0.0.1
10.0.0.2
10.0.0.3 **** failed ***
10.0.0.4 **** failed ***

If you'll ever find yourself in a situation where you'll need UDP flooding (serial line or device stress testing) but won't have a dedicated flood program available (they're usually just a few click away if you consult uncle Google), here's a Perl version of UDP flood:

#!/usr/bin/perl
##############

# udp flood.
##############
 
use Socket;
use strict;
 
if ($#ARGV != 3) {
  print "flood.pl <ip> <port> <size> <time>\n\n";
  print " port=0: use random ports\n";
  print " size=0: use random size between 64 and 1024\n";
  print " time=0: continuous flood\n";
  exit(1);
}
 
my ($ip,$port,$size,$time) = @ARGV;
 
my ($iaddr,$endtime,$psize,$pport);
 
$iaddr = inet_aton("$ip") or die "Cannot resolve hostname $ip\n";
$endtime = time() + ($time ? $time : 1000000);
 
socket(flood, PF_INET, SOCK_DGRAM, 17);

 
print "Flooding $ip " . ($port ? $port : "random") . " port with " . 
  ($size ? "$size-byte" : "random size") . " packets" . 
  ($time ? " for $time seconds" : "") . "\n";
print "Break with Ctrl-C\n" unless $time;
 
for (;time() <= $endtime;) {
  $psize = $size ? $size : int(rand(1024-64)+64) ;
  $pport = $port ? $port : int(rand(65500))+1;
 
  send(flood, pack("a$psize","flood"), 0, pack_sockaddr_in($pport, $iaddr));}

• Configuring Basic BGP
• Using Multihomed BGP Networks
• Employing AS - Path Filters
• Filtering with Prefix-Lists
• Influencing BGP Route Selection with Weights
• Implementing Changes in BGP Policy
• Configuring a Transit AS
• Introducing Route Reflectors
• Configuring and Monitoring Confederations
• Implementing BGP Peer Groups
• Setting BGP Local Preference
• Understanding BGP Multi-Exit Discriminator
• Addressing BGP Communities
• Limiting the Number of Prefixes Received from a BGP Neighbor
• Using BGP Route Dampening

If you're don't have access to Partner Education Connection, you can buy our Configuring BGP on Cisco Routers e-learning solution or the BGP Remote Lab Bundle.

In a previous post I've described how you can use neighbor local-as feature to fix AS-number mismatch between adjacent autonomous systems. However, without additional options, the local-as is inserted in the AS-path of incoming BGP updates before any inbound filters. Your inbound filters thus have to match the local-as as well.Consider, for example, the following configuration:

router bgp 65001
 neighbor 10.1.0.2 remote-as 10
 neighbor 10.1.0.2 local-as 20
 neighbor 10.1.0.2 filter-list 1 in
!
ip as-path access-list 1 permit ^10$

Although the configuration looks correct, no routes are accepted from AS 10, as the inbound AS-path always contains locally prepended AS 20 as well as AS 10:

R1#show ip bgp neighbor 10.1.0.2 received-routes | begin ^$
 
   Network     Next Hop      Metric LocPrf Weight Path
*  172.16.0.0  10.1.0.2           0             0 20 10 i

To fix this problem, you either have to include local AS in the AS-path access-list or use the no-prepend option of the neighbor local-as command.

You can get in-depth information on AS-path access-lists in our Configuring BGP on Cisco Routers e-learning solution. If you just need to practice them together with other BGP configuration tasks, the BGP Remote Lab Bundle is the perfect choice.

Marcus Jensen asked me a very interesting question:

I want to send 3 lines of configuration to a remote router, but I know the first line will kill my connection. Can I save these 3 lines of code to a text file, and then issue a Tcl command to add those to the running config?

The solution is much simpler and does not have to involve Tcl at all. The copy something system:running-config command merges the configuration commands in the source file with the current running configuration.

You can store the configuration commands you want to execute in a local file (even in NVRAM) or you could execute them directly off a file server (using HTTP, FTP, TFTP or SCP protocol).

This article is part of You've asked for it series.

If you're running NTP and SNTP on the same router, SNTP will never synchronize with a configured SNTP server, as the NTP process captures the reply packet before SNTP has the chance to react.

SNTP multicast/broadcast client mode works in combination with NTP

NTP process could be running even if your running configuration has no NTP-related commands. It starts automatically whenever you enter NTP-related configuration (ntp logging configuration command is enough) and is not stopped when the last NTP-related configuration command is removed. You have to reload the router to kill it.

On my test router, I've switched from NTP to SNTP by removing all NTP configuration and configuring SNTP server:

b2#show run | inc ntp
sntp logging
sntp server 192.168.0.5

However, SNTP would not synchronize as the NTP process was still running (check with the show processes | inc NTP command):

b2#show process | inc NTP
 259 Mwe 41E568BC   8  3233    2 3624/6000   0 NTP
 260 Mwe 4228BA9C   8    86   93 3660/6000   0 SNTP

NTP process was obviously receiving the NTP reply packets, as the debug sntp packet command did not display the return packets, even though the debug ntp packet peer command on the NTP server (192.168.0.5) displayed received and transmitted packets:

Sending SNTP packet to 192.168.0.5
 xmt CB66B3C0.8EC48A8E (14:12:16.557 UTC Wed Feb 20 2008)

After the reload, SNTP started working immediately and generated a syslog message indicating the time has been adjusted:

00:00:45: %SYS-6-CLOCKUPDATE: System clock has been updated from 15:00:59 UTC Wed Feb 20 2008 to 14:15:10 UTC Wed Feb 20 2008, configured from SNTP by 192.168.0.5.

The debugging printout also confirmed that the NTP reply packets were received by SNTP.

Sending SNTP packet to 192.168.0.5
 xmt CB66B4B8.D94D8B90 (14:16:24.848 UTC Wed Feb 20 2008)
Received SNTP packet from 192.168.0.5, length 48
 leap 0, mode 4, version 1, stratum 8, ppoll 16

 rtdel 00000C87, rtdsp 0000E5F1, refid 0A000005 (10.0.0.5)
 ref CB66B33F.2E02462D (14:10:07.179 UTC Wed Feb 20 2008)
 org CB66B4B8.D94D8B90 (14:16:24.848 UTC Wed Feb 20 2008)
 rec CB66B4B9.4C2B68CA (14:16:25.297 UTC Wed Feb 20 2008)
 xmt CB66B4B9.4C6FF670 (14:16:25.298 UTC Wed Feb 20 2008)
 inp CB66B4B8.DA2137A9 (14:16:24.852 UTC Wed Feb 20 2008)

Recently I had to create a new domain and the $1.88/year for an .info domain was simply irresistible ... so I also created the ioshints.info domain. The first results are already visible: my blog's URL has been changed to blog.ioshints.info. All redirections should be done transparently; if that's not the case, please let me know.

Next step: another round of negotiations with the hosting providers to get wiki.ioshints.info where I want it to be :)

In a previous post, I've described how you can detect high CPU load with the Embedded Resource Manager (ERM). If you want to respond to these events, you could use the syslog event detector within EEM, but it's more reliable to use the new event resource detector available in EEM version 2.2 (introduced in IOS release 12.4(2)T). The resource detector is best used in Tcl policy; if you use it in EEM applet, the same applet is triggered every time a resource policy threshold (minor/major/critical, rising or falling) is crossed. Within the EEM applet it's almost impossible to detect which threshold was crossed.

However, even EEM applet could solve some immediate problems. For example, if you want to store a snapshot of processes on a TFTP server every time the global CPU load crosses a policy threshold, you could use the following applet:

event manager applet ReportHighCPU
 event resource policy "HighGlobalCPU"
 action 1.0 cli command "show process cpu sorted 5sec | redirect tftp://10.0.0.10/highCPU$_resource_time_sent.txt"

To differentiate the snapshots, I've appended the _resource_time_sent variable set by the EEM before the applet is started to the file name, guaranteeing that the snapshot files will have unique names (at least until the router reload).

As an alternative, you could send the show process output in an e-mail:

event manager environment _ifDown_rcpt [email protected]
!
event manager applet ReportHighCPU
 event resource policy "HighGlobalCPU"
 action 1.0 cli command "show process cpu sorted 5sec"
 action 1.1 info type routername
 action 2.0 mail server "mail-gw" →
    to "$_ifDown_rcpt" from "[email protected]" →
    subject "CPU @ $_resource_current_value" →
    body "$_cli_result"

This article is part of You've asked for it series.

A few days ago, I've published a link to a series of OSPF-over-WAN tutorials by Arden Packeer. He went a step further and analyzed what happens if you mix supposedly incompatible OSPF network types in the same IP subnet.

Mike Nipp has wondered which syslog message to use to reliably detect router reload under all circumstances:

The problem I had with the SYS-5-RESTART message is I don't think you will get one if the power is suddenly pulled from the router. It does do a SNMP-5-COLDSTART and SYS-6-BOOTTIME on boot up.

I did an actual power-cycle test of a router (we can do that remotely in our labs, so I didn't have to touch the box :) and the SYS-5-RESTART message is reliably generated at every startup, be it from the power cycle or the reload command (I was not able to provoke an on-demand crash ;).

This article is part of You've asked for it series.

The Scale your backbone with core MPLS, BGP on the edge article I wrote for SearchTelecom describes how you can improve the stability and scalability of a Service Provider core network by replacing BGP on the core routers with MPLS switching. You'll also find a typical design scenario and a few recommendations that you should follow to prevent router reloads from introducing black holes in your MPLS network.

The list of all articles I wrote for SearchTelecom is available in the CT3 wiki.

The post by Ethan Banks reminded me of a “mystery” I was trying to solve years ago when developing my first BGP course. The Cisco documentation has always claimed there were four well-known communities (the Internet community being one of them), while the RFC 1997 lists three well-known values. Unfortunately, most people blindly copy the IOS documentation (including the authors of the latest revision of the Cisco’s BGP course) without asking themselves “what the heck is the Internet community”.

I don’t remember when exactly I’ve created the BGP community chapter of that BGP course, but I was able to fetch a very old BGP course description from the Internet Archives … and by that time, the course was in its fifth or sixth revision. It must have been 10 years ago.

It was time to revisit the mystery. I’ve tried applying the Internet community to a network originated by the BGP routing process to see what its value is:

router bgp 65000
 network 192.168.1.0 route-map SetInternet
!
route-map SetInternet
 set community internet additive

While the router obediently attached the Internet community to the IP prefix, I was no wiser … all show outputs converted the community value into its symbolic name. I had to use Wireshark and analyze the actual routing updates between BGP neighbors to figure out that the Internet community has an illegal value 0:0. Obviously it’s not a well-known community.

If you're looking for more in-depth BGP knowledge, try our Configuring BGP on Cisco Routers e-learning solution. If you just need to enhance your hands-on skill, the BGP Remote Lab Bundle is the perfect choice.

Digging through old materials finally gave me the answer I was looking for: sometimes you need a permit all at the end of the ip community-list (like access-lists, the community-lists have an implicit deny all at the end) and someone decided that permit internet makes more sense than the familiar permit any (yes, that’s correct … you use the keyword internet to match any community in the ip community-list).

And just for the sake of completeness, let me conclude with a ten year old slide explaining this phenomenon:

Cisco partners and employees can access the BGP Communities remote lab free-of-charge on the Partner Education Connection.

David Winter wanted to detect high-CPU spikes and act on them. The first part (high CPU utilization) could be done with SNMP, but since IOS release 12.3(14)T, the right tool for the job is the Embedded Resource Manager (ERM).

The ERM syntax is a bit baroque (and not well documented), so let's work through the example: this is the configuration you need to detect high overall CPU utilization on the main CPU in the box:

resource policy
 policy HighGlobalCPU global
  system
   cpu total
    critical rising 95 falling 70 interval 10
    major rising 75 falling 50 interval 10
 !
 user global HighGlobalCPU

And here are the usage/configuration guidelines:

• The whole ERM subsystem is configured under the resource policy section;
• You always have to configure a policy and a user to which the policy applies. In our example, the user is global (as we're measuring the global CPU load);
• The policy we're defining must have the global keyword to indicate we're measuring overall utilization (otherwise you can't attach it to the global user);
• We're measuring the load on the main CPU, so we're configuring the system subsection of the policy (on distributed platforms you could specify slot name to measure utilization on a specific linecard);
• The cpu section selects CPU load measurements. You could measure interrupt load, process load or total CPU load.
• Within each resource section in the policy (in our example, total CPU load on the main system) you can define minor, major and critical thresholds (syslog messages are generated when each threshold is crossed).
• After the policy is defined, it's applied to the global user.

With the CPU load measurement policy defined, the router will generate syslog messages (SYS-4-CPURESRISING) every time the overall CPU load exceeds the specified rising thresholds. When the utilization falls below the falling threshold, the SYS-4-CPURESFALLING syslog message is generated.

This article is part of You've asked for it series.

In IOS release 12.4(15)T (and potentially in all older releases, I've tested it with 12.4(11)T4), the route-map option of the neighbor fall-over command that I've described in my March IP Corner article, Designing Fast Converging BGP Networks, is not saved in the startup configuration. You have to reconfigure the BGP neighbors after each router reload.

Furthermore, a BGP neighbor does not inherit the fall-over route-map option from a peer session template.

Both issues are fixed in 12.4(15)T3.

I'm probably getting old … I keep forgetting the exact names (and capitalization) of route-maps I've configured on the router. The show route-maps command is way too verbose when I'm simply looking for the exact name of the route-map I want to use, so I wrote a Tcl script that displays the names of the route-maps configured on the router. If you add a -d switch, it also displays their descriptions (to be more precise, the first description configured in the route-map).

When using the -d switch, the script executes the show running command and might take a while to complete.

To use the script, download the routeMaps.tcl file (available from my web site) into the router's flash and follow the installation instructions in the source.

Here is a sample printout from one of my routers:

R1#show alias | include rm

  rm                    tclsh flash:routeMaps.tcl
R1#rm
LocPref
SetCommunity
TestRange
prepend
 
R1#rm -d
Route map name       Description
========================================================================
LocPref
SetCommunity         Sets time-based communities on local routes
TestRange
prepend

cpmf14 has left an interesting comment documenting how to perform a periodic back up of a file in router's flash to an FTP server:

event manager applet backup-crl
 event timer watchdog time 86400 maxrun 4294967295
 action 1.0 cli command "enable"
 action 2.0 cli command "copy flash:/iosca.crl ftp://username:[email protected]/" pattern "a.b.c.d"
 action 3.0 cli command "a.b.c.d" pattern "iosca.crl"
 action 4.0 cli command "iosca.crl"
 action 5.0 syslog msg "FTP backup successful"

We all know that BGP responds very slowly to changes in network topology, right? Wrong! When Cisco's engineers fixed the IOS BGP code, they dramatically improved BGP's response times. In the March IP Corner article, Designing Fast Converging BGP Networks, I'm describing one of the new features, the BGP fast peering session deactivation, including in-depth description of how it works, design guidelines and recently added options that are not well explained in Cisco's documentation (one of the features is so new it's been working properly only since 12.4(15)T3).

If you want to match IP address as well as the subnet mask of a BGP route, you can use extended IP access-lists to match both. The extended access-lists can be used in neighbor distribute-list in/out router configuration command or in a match ip address command within a route-map.

When I've included a few slides on this feature in the first BGP course I've developed for Cisco (that was probably somewhere around 1994), the results in the class were always the same: total confusion that needed an hour of whiteboard examples to dissolve. You can find a few examples that will help you understand this arcane feature in a post written by Brian Dennis.

The use of extended IP ACL as a route matching mechanism was made obsolete by the ip prefix-list command, which was introduced in 12.0T. As 12.0T reached End-of-Engineering in the previous millennium, it's a safe bet that the only place where you might still be required to use extended ACLs to match IP routes is in the CCIE lab.

In response to my January IP Corner article The Never-Ending Story of IP Fragmentation, Stojanco Cavdarov made an interesting observation: routers are not allowed to fragment IPv6 packets, they have to respond back with ICMP unreachable (effectively, routers behave as if IPv6 packets would have an implicit don't fragment bit).

To make life easier for non-TCP IPv6 applications (TCP is supposed to use Path MTU Discovery), the minimum IPv6 packet size that has to be supported on all links was increased to 1280 bytes (which, incidentally, fits very nicely into GRE+IPSec envelope transported across links with 1500-byte MTU).

Whisper asked an interesting question …

“What I would like to know is, on my PPP negotiated ADSL connection, how the ISP assigns me a /32 ip address.”

… which prompted me to test various WAN encapsulations and address assignment rules. Here are the results:

• On all WAN encapsulations you can configure subnet masks down to /31 (/30 in old IOS releases).
• The same IP address can be used on more than one interface as long as both IP address and subnet mask match.
• Two WAN interfaces can have different IP addresses but still belong to the same IP subnet. You would use this on Frame Relay when you have multiple interfaces into the same FR cloud for bandwidth reasons.
• If you configure IP address with IPCP (with the ip address negotiated command), the subnet mask becomes /32 as IPCP does not carry subnet mask (and you get the host route toward the PPP peer unless you turn off the PPP peer route option)
• If you configure IP address with SLARP (Serial Line ARP) on HDLC, the subnet mask is inherited from the peer (HDLC SLARP carries subnet mask) and the IP address is determined by flipping the low-order bits in the neighbor's IP address.

This article is part of You've asked for it series.

In a previous post, I've described how you can use Tcl shell to upload text content into router's flash if the router has no connectivity to a suitable file server (or you don't have FTP or TFTP server handy).

The trick works flawlessly, but typing the same obscure Tcl commands gets tedious after a while, so the first time I had to use this solution to develop a Tcl script, I've quickly written another script that takes file name as the parameter and hides all the other murky details.

To use it, transfer the contents of storeFile.tcl (available from my web site) to the router's flash (using the previously described trick), follow the installation instructions in the source and you're ready to go.

Note: You can adapt the Tcl script to your needs; for example, you could add instructions to re-register EEM Tcl policy every time you upload the new code.

Debugging time-based configurations could be a nightmare, as you have to switch router's time back and forth trying to debug your configuration and wait for the desired event to occur. When I was debugging my EEM-based solution to time-based BGP policy routing, I simply defined two aliases that would set the clock to 30 seconds before the event I wanted to test:

alias exec 859 clock set 08:59:30
alias exec 900 clock set 09:00:30

Obviously, these tests are best done in a lab setup … and you have to turn off NTP or any other form of time synchronization.