Blog Posts in October 2007

Remove the configuration prompt

I should probably write this one on April 1st, but maybe October 31st is not such a bad choice after all … if you configure no service prompt config, the configuration prompt is gone; when you enter the configuration mode with the configure terminal command, you get an empty line (like you did with Cisco software release 9.1 some 15 years ago). Similarly, you can disable command-line editing with the no editing line configuration command or terminal no editing exec-level command. If only there would be a way to disable the context-sensitive help :)
see 9 comments

More Details on OSPF Route Filters

I did a few follow-up tests with the distribute-list in OSPF configuration command and stumbled across a few interesting facts (IOS release 12.4(15)T1 on a 3725 platform):

  • Although the router allows you to configure distribute-list acl in interface, it does not work. Routes received through that interface (or having the interface as the next-hop) are not filtered.
  • When you apply the distribute-list in command, the routing table is not changed. Clearing the IP routing table does not help, you have to clear ALL OSPF processes (including bringing down all OSPF adjacencies) with the clear ip ospf process command for the route filter to take effect.
  • The same limitations don't apply in the other direction: when you remove the distribute-list in, SPF is triggered and the routes appear in the IP routing table automatically.
  • The somewhat undocumented gateway option of the distribute-list in command works, but not quite as I would expect: the IP next hop, not the router-ID of the router advertising the IP prefix is matched by the prefix-list.

And, last but not least, I've lab-verified my previous claim: applying the distribute-list in on a transit router can result in a black hole, as the LSAs themselves are not filtered.

see 9 comments

Send an e-mail when an interface goes down

John S. Pumphrey recently asked an interesting question: “Can the router send an e-mail when an interface goes down?” The enterprisey solution is obvious: deploy a high-end EMS to collect SNMP traps and use its API to write a custom module that would use a MQ interface to alert the operator. Fortunately, Event Manager applets in Cisco IOS provide action mail command (available in 12.3(14)T and 12.4) that can send an e-mail to a SMTP server straight from the router.

There are two ways you can detect that an interface went down with EEM: either you track the interface status with a track object and start an EEM applet when the track object changes state or you catch the syslog messages reporting that the interface line protocol changed state to down. The second approach is obviously more generic, as a single applet can act on multiple interfaces.

event manager applet MailOnIfDown
 event syslog occurs 1 →
    pattern "LINEPROTO-5-UPDOWN.*to down" →
    period 1


  • If you want to limit the applet to serial interfaces only, you could change the pattern to LINEPROTO-5-UPDOWN.*Serial.*to down.
  • The → continuation character is used to indicate that a single configuration line has been split to increase readability.

The action mail command specifies the mail server's address (use a hostname and DNS lookup or ip host configuration command to make the EEM applet more generic), from and to address, message subject and body. In each of these fields, you can use EEM environment variables that you can define with the event manager environment configuration command. Each EEM event also defines a few environment variables that you can use (see the table of EEM system-defined variables on CCO). For example, you can define the e-mail recipient in the router's configuration and use the _syslog_msg variable to include the syslog message in the e-mail body:

event manager environment _ifDown_rcpt [email protected]
event manager applet MailOnIfDown
 event syslog occurs 1 →
    pattern "LINEPROTO-5-UPDOWN.*to down" →
    period 1
 action 1.0 mail server "mail-gw" →
    to "$_ifDown_rcpt" from "[email protected]" →
    subject "Interface down on R1" →
    body "$_syslog_msg"

You can make the applet even more generic with the help of action info type routername command, which stores the current router's name into the $_info_routername environment variable:

event manager environment _ifDown_rcpt [email protected]
event manager applet MailOnIfDown
 event syslog occurs 1 →
    pattern "LINEPROTO-5-UPDOWN.*to down" →
    period 1
 action 1.0 info type routername
 action 2.0 mail server "mail-gw" →
    to "$_ifDown_rcpt

" from "$" →
    subject "Interface down on $_info_routername" →
    body "$_syslog_msg"

Note: This article is part of You've asked for it series.

see 3 comments

Debugging cached CEF adjacencies

A while ago I wrote about cached CEF adjacencies and the impact they have on ARP caching. If you ever need to, you can debug them with the debug ip cef table command. As this command might produce a lot of output in a production network, always use it in combination with an access-list that limits the debugging to the selected address range.

Alternatively, you can use the debug arp adjacency command, but you cannot limit its output with an access-list

read more add comment

GRE tunnel keepalives

The IP-over-IP (usually GRE) tunnels (commonly in combination with IPSec to provide security) are frequently used when you want to transport private IP traffic over public IP network that does not support layer 3 VPNs. If you use the GRE tunnels in combination with default routing (or route summarization), you can get serious routing issues when the tunnel destination disappears, but a default (or summary) route in the IP routing table still covers it. You could work around this issue by deploying a routing protocol over the GRE tunnel (which could lead to hard to diagnose routing loops if you're not careful) or by using GRE keepalives introduced in IOS release 12.2(8)T.

The implementation of the GRE keepalives is amazing: the router sending the keepalive packet constructs a GRE packet that would be sent from the remote end back to itself (effectively building a GRE reply), sets the GRE protocol type to zero (to indicate the keepalive packet) and sends the whole packet through the tunnel (effectively encapsulating GRE reply into another GRE envelope). The receiving router strips the GRE envelope and routes the inside packet … which is the properly formatted GRE keepalive reply.

This trick allows you to implement different GRE keepalive timers on each end of the link. For example, the remote site might use fast keepalive timers to detect loss of primary link and switch over to a backup link, while the central site would use less frequent keepalive tests to detect failed remote site (if there is a single path to the remote site, you don't care too much when you detect it's down).

Every ingenious solution has its drawbacks and this one is no exception: if the receiving router protects its IP addresses (to stop spoofing attacks), it will drop the incoming GRE keepalive packet. Furthermore, a document available on Cisco's web describes the issues of using GRE keepalives in IPSec environment.
see 12 comments

Tabular display of interface MTUs

When I started exploring the details of MTU handling in Cisco IOS, I quickly got tired of analyzing various long printouts to extract the MTU sizes, so I wrote a Tcl script that display hardware, IP and MPLS MTUs in a tabular format. To install it on your router:
  1. Download it from my web site and copy it to your router's flash or NVRAM.
  2. Define an alias, for example alias exec mtu tclsh flash:displayMTU.tcl.

The script recognizes two parameters: the ip parameter displays only the interfaces that have IP configured and the mpls parameter displays only the MPLS-enabled interfaces.

read more see 2 comments

The tale of the three MTUs

An IOS device configured for IP+MPLS routing uses three different Maximum Transmission Unit (MTU) values:

  • The hardware MTU configured with the mtu interface configuration command
  • The IP MTU configured with the ip mtu interface configuration command
  • The MPLS MTU configured with the mpls mtu interface configuration command

The hardware MTU specifies the maximum packet length the interface can support … or at least that's the theory behind it. In reality, longer packets can be sent (assuming the hardware interface chipset doesn't complain); therefore you can configure MPLS MTU to be larger than the interface MTU and still have a working network. Oversized packets might not be received correctly if the interface uses fixed-length buffers; platforms with scatter/gather architecture (also called particle buffers) usually survive incoming oversized packets.

IP MTU is used to determine whether a non-labeled IP packet forwarded through an interface has to be fragmented (the IP MTU has no impact on labeled IP packets). It has to be lower or equal to hardware MTU (and this limitation is enforced). If it equals the HW MTU, its value does not appear in the running configuration and it tracks the changes in HW MTU. For example, if you configure ip mtu 1300 on a Serial interface, it will appear in the running configuration as long as the hardware MTU is not equal to 1300 (and will not change as the HW MTU changes). However, as soon as the mtu 1300 is configured, the ip mtu 1300 command disappears from the configuration and the IP MTU yet again tracks the HW MTU.

The MPLS MTU determines the maximum size of a labeled IP packet (MPLS shim header + IP payload size). If the overall length of the labeled packet (including the shim header) is greater than the MPLS MTU, the packet is fragmented. The MPLS MTU can be greater than the HW MTU assuming the hardware architecture and interface chipset support that (and the router will warn you that you might be getting into trouble). Similar to the ip mtu command, the mpls mtu command will only appear in the running configuration if the MPLS MTU is different from the HW MTU. However, contrary to the behavior of the IP MTU, any change in HW MTU with the mtu configuration command also resets the MPLS MTU to HW MTU.

The behavior as described above was tested on a 3725 router running IOS release 12.4(15)T1. Although the MPLS MTU Command Changes document claims that you cannot set MPLS MTU larger than then interface MTU from IOS release 12.4(11)T, I was still able to do it in 12.4(15)T1.

see 13 comments

SNMP with Tcl

Looking from the outside, it looks like Tcl SNMP routines in Cisco IOS were designed by a commitee or came straight from Dilbert. The snmp_getone function that reads a single SNMP value does not return an array or a list (as one would expect), but a string representation of something that looks like an XML object (but is not, since its attributes are not properly quoted). As Tcl on Cisco IOS has no built-in XML support, parsing the return values is a pure joy (and a nice exercise in writing regular expressions).

The following excerpt of a telnet session shows how to extract a single SNMP value in Tcl (I've used extra steps and an interactive tclsh session for illustration purposes). The SNMP community has to be configured in advance with the snmp-server community test ro configuration command.
rtr(tcl)#set value [snmp_getone test system.3.0]
{<obj oid='sysUpTime.0' val='14886'/>}
rtr(tcl)#regexp -inline {oid='(.*)'.*val='(.*)'} $value
{oid='sysUpTime.0' val='14886'} sysUpTime.0 14886
rtr(tcl)#regexp {oid='(.*)'.*val='(.*)'} $value ignore oid result
rtr(tcl)#puts $result
And now for a complete example: the following script prints the router uptime.
# Simple Tcl script to print system uptime
set value [snmp_getone test system.3.0]
regexp {oid='(.*)'.*val='(.*)'} $value ignore oid result
set result [expr $result / 100]
puts "Router uptime is $result seconds"
see 3 comments

War story: almost zero is not good enough

Some fifteen years ago we were building a router-based network using primarily baseband modems (that's how the DSL boxes with symmetrical speeds were called back then). Everything worked great, we even had DECnet running between a few sites. However, after a few weeks, a mystery phenomena crept up: when the users were copying files between two VAX computers, the link between the sites went down … always when copying the same file.

To make the long story short: every modem uses a predefined sequence that enables remote loopback. These sequences are standardized and have been chosen so that the chance of hitting them with real traffic is close to zero (but obviously not zero). The file that the users were trying to copy contained just the correct sequence to trigger the remote loopback in one of the modems and the link went down (most probably changed state to looped) as the routers started to receive their own packets. Disabling remote loopback on the modem (a jumper in those days) solved the problem.

The moral of the story: whenever you use pattern matching to identify something (be it a specific application that you're trying to identify or a virus in your workstation), there is a non-zero chance of false positives, usually in the most unusual places.

Let's conclude this post with what seems to me a ludicrous “invention”: someone patented the flashing of LEDs when performing loopback tests on a modem. If anyone can survive reading the whole patent application, understand it and recognize its true added value, please let me know … I got completely lost.
see 4 comments

Turn your flash card into an ATA drive

The flash memory available in newer router platforms (at the very minimum the ISR routers and 37xx series) is capable of being used as a regular disk drive (for example, to store system logging information), but it might be formatted as a traditional Low-End File System (LEFS) flash card (more likely if the router was not manufactured recently). To change the flash card format to disk-like FAT32 format, use the format flash: privileged-level command (and don't forget to store the IOS image to another location before formatting the flash). After the format process is complete, you can create subdirectories on the flash: memory and use it as a regular disk device.
read more see 1 comments

mturoute: trace mode output

Continuing from the previous mturoute-related post, this is how the mturoute utility behaves when you start it in traceroute mode (with the -t flag):
  • Similar to Windows tracert, it tries to find the successive hops in the path by sending ICMP echo packets with increasing values of TTL field.
  • Contrary to Cisco IOS and most Unix systems that send UDP packets to high-numbered ports, tracert uses ICMP echo packets.

  • For each router found in the path (= source IP address in the ICMP TTL exceeded message), mturoute tries to find path MTU to that hop using the same algorithm as in the ping mode.
  • During the bisecting phase, the mturoute does not print all the messages it prints in the ping mode, but just the cryptic signs (+/-/u/.) indicating its progress. Their meaning is documented in the previous post.
  • After the path MTU to the router under investigation is measured, mturoute reports the router's IP address and path MTU.
read more add comment

Display IP packet filters attached to router's interfaces

A few days ago, Jeremy Stretch asked me whether there's a command to display packet lists attached to router's interfaces. While he got pretty far with the output filters, he would like to have a nice tabular format as well as the contents of the access lists displayed next to the interfaces. The show ip access-list interface name command comes pretty close, but it displays the information only for a single interface, so it was time to write another Tcl script. To install it on your router:
  1. Download it from my web site and copy it to your router's flash or NVRAM.
  2. Define an alias, for example alias exec filters tclsh flash:packetFilters.tcl.

The script recognizes two parameters: the all parameter displays all interfaces, including ones with no access lists and the verbose parameter displays the contents of the access list after the interface name.

read more see 4 comments

IOS 12.4T features summarized on one page

I always thought that the new format of the Cisco web-based documentation was awful, as it consumes way more bandwidth than the old version and is slower to load over low-speed links as it displays the text only after the complete page is loaded due to heavy use of table-based HTML layout (I will refrain from commenting the use of this layout technique in the third millenium). However, the new content structure has some significant benefits; for example, all the 12.4T feature guides are collected on a single page … fantastic if you try to find a feature that you remember was implemented somewhere in 12.4T track.
see 1 comments

mturoute: ping-mode output

Jeff West has asked me to document the printout produced by the mturoute utility. Here's the first part of the documentation.

mturoute works in two modes:
  • Without the -t flag, it sends variable-lenght ICMP echo packets to the specified destination address, trying to figure out the largest packet that is successfully propagated to the destination.
  • With the -t flag, it uses traceroute-like algorithm to find the hop-by-hop IP addresses (the source IP addresses of the ICMP TTL exceeded replies) and uses the same packet-size-calculating algorithm to measure the path MTU to each hop.

Today we'll focus on the non-trace mode. It tries to measure the path MTU with a bisection method varying the packet sizes between minimum MTU (92) and maximum MTU (specified with the -m parameter, default is 10000 bytes). The payload size of the first packet (without the -m flag) is thus 5046 bytes ((10000 + 92)/2).

On each iteration, the algorithm prints a “cryptic” sign indicating whether the ping with the current payload size succeeded. The following indications are given:

  • '+': ICMP echo reply arrived
  • '-': The ping failed (for various reasons, including exceeding the path MTU)
  • 'u': ICMP destination unreachable response arrived, indicating blackhole or access-list.
  • ICMP unreachable is considered a successful response; at least we're measuring the path MTU up to the failure point

  • '.': timeout. The ping will be retried up to three times (or the number specified with the -r option).
In the ping mode, mturoute prints the current ICMP payload size at each step, resulting in a printout similar to the one below. If you'd have specified the -d option, the printout would include detailed status codes from the IcmpSendEcho function.
$ mturoute
* ICMP Fragmentation is not permitted. *
* Maximum payload is 10000 bytes. *
- ICMP payload of 5046 bytes failed..
- ICMP payload of 2569 bytes failed..
+ ICMP payload of 1330 bytes succeeded.
- ICMP payload of 1949 bytes failed..
- ICMP payload of 1639 bytes failed..
- ICMP payload of 1484 bytes failed..
+ ICMP payload of 1407 bytes succeeded.
- ICMP payload of 1445 bytes failed..
+ ICMP payload of 1426 bytes succeeded.
+ ICMP payload of 1435 bytes succeeded.
+ ICMP payload of 1440 bytes succeeded.
+ ICMP payload of 1442 bytes succeeded.
+ ICMP payload of 1443 bytes succeeded.
+ ICMP payload of 1444 bytes succeeded.
+ ICMP payload of 1444 bytes succeeded.
Path MTU: 1472 bytes.

Note: To use the debug-enabled version of mturoute, or the version that does not need VC++ runtime, download the new ZIP archive from my web site.

see 3 comments

Show IP access lists attached to an interface

When developing yet another Tcl script, I've stumbed across an interesting show command: the show ip access-list interface name introduced in IOS release 12.4(6)T displays the contents of the inbound and outbound IP access-list applied to the specified interface. The really nice part is that the ACL statistics (number of matches displayed next to the ACL lines) are kept and displayed per-interface.
read more see 1 comments

OSPF Router-Id Does Not Change When the Interface IP Address Changes

The venerable rules used to establish OSPF router ID on Cisco IOS are all over the Internet:

  • Take the highest IP address of all loopback interfaces configured on the router when the OSPF process is started.
  • If there is no loopback interface, take the highest IP address of an operating interface.

In the old days, when Cisco believed that the router ID had to match an interface address, this also implied that the router ID would have changed if the interface IP address changed (and we told the students that you have to use loopback interfaces to make your network stable, as the OSPF process would restart if the interface giving the router ID went down).

read more see 3 comments