Blog Posts in October 2007
Remove the configuration prompt
More Details on OSPF Route Filters
I did a few follow-up tests with the distribute-list in OSPF configuration command and stumbled across a few interesting facts (IOS release 12.4(15)T1 on a 3725 platform):
- Although the router allows you to configure distribute-list acl in interface, it does not work. Routes received through that interface (or having the interface as the next-hop) are not filtered.
- When you apply the distribute-list in command, the routing table is not changed. Clearing the IP routing table does not help, you have to clear ALL OSPF processes (including bringing down all OSPF adjacencies) with the clear ip ospf process command for the route filter to take effect.
- The same limitations don't apply in the other direction: when you remove the distribute-list in, SPF is triggered and the routes appear in the IP routing table automatically.
- The somewhat undocumented gateway option of the distribute-list in command works, but not quite as I would expect: the IP next hop, not the router-ID of the router advertising the IP prefix is matched by the prefix-list.
And, last but not least, I've lab-verified my previous claim: applying the distribute-list in on a transit router can result in a black hole, as the LSAs themselves are not filtered.
Send an e-mail when an interface goes down
John S. Pumphrey recently asked an interesting question: “Can the router send an e-mail when an interface goes down?” The enterprisey solution is obvious: deploy a high-end EMS to collect SNMP traps and use its API to write a custom module that would use a MQ interface to alert the operator. Fortunately, Event Manager applets in Cisco IOS provide action mail command (available in 12.3(14)T and 12.4) that can send an e-mail to a SMTP server straight from the router.
There are two ways you can detect that an interface went down with EEM: either you track the interface status with a track object and start an EEM applet when the track object changes state or you catch the syslog messages reporting that the interface line protocol changed state to down. The second approach is obviously more generic, as a single applet can act on multiple interfaces.
event manager applet MailOnIfDown
event syslog occurs 1 →
pattern "LINEPROTO-5-UPDOWN.*to down" →
period 1
Notes:
- If you want to limit the applet to serial interfaces only, you could change the pattern to LINEPROTO-5-UPDOWN.*Serial.*to down.
- The → continuation character is used to indicate that a single configuration line has been split to increase readability.
The action mail command specifies the mail server's address (use a hostname and DNS lookup or ip host configuration command to make the EEM applet more generic), from and to address, message subject and body. In each of these fields, you can use EEM environment variables that you can define with the event manager environment configuration command. Each EEM event also defines a few environment variables that you can use (see the table of EEM system-defined variables on CCO). For example, you can define the e-mail recipient in the router's configuration and use the _syslog_msg variable to include the syslog message in the e-mail body:
event manager environment _ifDown_rcpt [email protected]
!
event manager applet MailOnIfDown
event syslog occurs 1 →
pattern "LINEPROTO-5-UPDOWN.*to down" →
period 1
action 1.0 mail server "mail-gw" →
to "$_ifDown_rcpt" from "[email protected]" →
subject "Interface down on R1" →
body "$_syslog_msg"
You can make the applet even more generic with the help of action info type routername command, which stores the current router's name into the $_info_routername environment variable:
event manager environment _ifDown_rcpt [email protected]" from "$_info_routername@lab.com" →
!
event manager applet MailOnIfDown
event syslog occurs 1 →
pattern "LINEPROTO-5-UPDOWN.*to down" →
period 1
action 1.0 info type routername
action 2.0 mail server "mail-gw" →
to "$_ifDown_rcpt
subject "Interface down on $_info_routername" →
body "$_syslog_msg"
Note: This article is part of You've asked for it series.
Debugging cached CEF adjacencies
Alternatively, you can use the debug arp adjacency command, but you cannot limit its output with an access-list
OSPF Neighbors Stuck in EXSTART
This problem is rare but tantalizing enough to warrant mentioning: OSPF neighbors are forever stuck in the EXSTART state (occasionally going DOWN and back to EXSTART).
GRE tunnel keepalives
The implementation of the GRE keepalives is amazing: the router sending the keepalive packet constructs a GRE packet that would be sent from the remote end back to itself (effectively building a GRE reply), sets the GRE protocol type to zero (to indicate the keepalive packet) and sends the whole packet through the tunnel (effectively encapsulating GRE reply into another GRE envelope). The receiving router strips the GRE envelope and routes the inside packet … which is the properly formatted GRE keepalive reply.
This trick allows you to implement different GRE keepalive timers on each end of the link. For example, the remote site might use fast keepalive timers to detect loss of primary link and switch over to a backup link, while the central site would use less frequent keepalive tests to detect failed remote site (if there is a single path to the remote site, you don't care too much when you detect it's down).
Every ingenious solution has its drawbacks and this one is no exception: if the receiving router protects its IP addresses (to stop spoofing attacks), it will drop the incoming GRE keepalive packet. Furthermore, a document available on Cisco's web describes the issues of using GRE keepalives in IPSec environment.
Tabular display of interface MTUs
- Download it from my web site and copy it to your router's flash or NVRAM.
- Define an alias, for example alias exec mtu tclsh flash:displayMTU.tcl.
The script recognizes two parameters: the ip parameter displays only the interfaces that have IP configured and the mpls parameter displays only the MPLS-enabled interfaces.
The tale of the three MTUs
An IOS device configured for IP+MPLS routing uses three different Maximum Transmission Unit (MTU) values:
- The hardware MTU configured with the mtu interface configuration command
- The IP MTU configured with the ip mtu interface configuration command
- The MPLS MTU configured with the mpls mtu interface configuration command
The hardware MTU specifies the maximum packet length the interface can support … or at least that's the theory behind it. In reality, longer packets can be sent (assuming the hardware interface chipset doesn't complain); therefore you can configure MPLS MTU to be larger than the interface MTU and still have a working network. Oversized packets might not be received correctly if the interface uses fixed-length buffers; platforms with scatter/gather architecture (also called particle buffers) usually survive incoming oversized packets.
IP MTU is used to determine whether a non-labeled IP packet forwarded through an interface has to be fragmented (the IP MTU has no impact on labeled IP packets). It has to be lower or equal to hardware MTU (and this limitation is enforced). If it equals the HW MTU, its value does not appear in the running configuration and it tracks the changes in HW MTU. For example, if you configure ip mtu 1300 on a Serial interface, it will appear in the running configuration as long as the hardware MTU is not equal to 1300 (and will not change as the HW MTU changes). However, as soon as the mtu 1300 is configured, the ip mtu 1300 command disappears from the configuration and the IP MTU yet again tracks the HW MTU.
The MPLS MTU determines the maximum size of a labeled IP packet (MPLS shim header + IP payload size). If the overall length of the labeled packet (including the shim header) is greater than the MPLS MTU, the packet is fragmented. The MPLS MTU can be greater than the HW MTU assuming the hardware architecture and interface chipset support that (and the router will warn you that you might be getting into trouble). Similar to the ip mtu command, the mpls mtu command will only appear in the running configuration if the MPLS MTU is different from the HW MTU. However, contrary to the behavior of the IP MTU, any change in HW MTU with the mtu configuration command also resets the MPLS MTU to HW MTU.
The behavior as described above was tested on a 3725 router running IOS release 12.4(15)T1. Although the MPLS MTU Command Changes document claims that you cannot set MPLS MTU larger than then interface MTU from IOS release 12.4(11)T, I was still able to do it in 12.4(15)T1.
SNMP with Tcl
The following excerpt of a telnet session shows how to extract a single SNMP value in Tcl (I've used extra steps and an interactive tclsh session for illustration purposes). The SNMP community has to be configured in advance with the snmp-server community test ro configuration command.
rtr#tclshAnd now for a complete example: the following script prints the router uptime.
rtr(tcl)#set value [snmp_getone test system.3.0]
{<obj oid='sysUpTime.0' val='14886'/>}
rtr(tcl)#regexp -inline {oid='(.*)'.*val='(.*)'} $value
{oid='sysUpTime.0' val='14886'} sysUpTime.0 14886
rtr(tcl)#regexp {oid='(.*)'.*val='(.*)'} $value ignore oid result
1
rtr(tcl)#puts $result
14886
#
# Simple Tcl script to print system uptime
#
set value [snmp_getone test system.3.0]
regexp {oid='(.*)'.*val='(.*)'} $value ignore oid result
set result [expr $result / 100]
puts "Router uptime is $result seconds"
War story: almost zero is not good enough
To make the long story short: every modem uses a predefined sequence that enables remote loopback. These sequences are standardized and have been chosen so that the chance of hitting them with real traffic is close to zero (but obviously not zero). The file that the users were trying to copy contained just the correct sequence to trigger the remote loopback in one of the modems and the link went down (most probably changed state to looped) as the routers started to receive their own packets. Disabling remote loopback on the modem (a jumper in those days) solved the problem.
The moral of the story: whenever you use pattern matching to identify something (be it a specific application that you're trying to identify or a virus in your workstation), there is a non-zero chance of false positives, usually in the most unusual places.
Let's conclude this post with what seems to me a ludicrous “invention”: someone patented the flashing of LEDs when performing loopback tests on a modem. If anyone can survive reading the whole patent application, understand it and recognize its true added value, please let me know … I got completely lost.
Turn your flash card into an ATA drive
mturoute: trace mode output
- Similar to Windows tracert, it tries to find the successive hops in the path by sending ICMP echo packets with increasing values of TTL field.
- For each router found in the path (= source IP address in the ICMP TTL exceeded message), mturoute tries to find path MTU to that hop using the same algorithm as in the ping mode.
- During the bisecting phase, the mturoute does not print all the messages it prints in the ping mode, but just the cryptic signs (+/-/u/.) indicating its progress. Their meaning is documented in the previous post.
- After the path MTU to the router under investigation is measured, mturoute reports the router's IP address and path MTU.
Contrary to Cisco IOS and most Unix systems that send UDP packets to high-numbered ports, tracert uses ICMP echo packets.
Display IP packet filters attached to router's interfaces
- Download it from my web site and copy it to your router's flash or NVRAM.
- Define an alias, for example alias exec filters tclsh flash:packetFilters.tcl.
The script recognizes two parameters: the all parameter displays all interfaces, including ones with no access lists and the verbose parameter displays the contents of the access list after the interface name.
IOS 12.4T features summarized on one page
mturoute: ping-mode output
mturoute works in two modes:
- Without the -t flag, it sends variable-lenght ICMP echo packets to the specified destination address, trying to figure out the largest packet that is successfully propagated to the destination.
- With the -t flag, it uses traceroute-like algorithm to find the hop-by-hop IP addresses (the source IP addresses of the ICMP TTL exceeded replies) and uses the same packet-size-calculating algorithm to measure the path MTU to each hop.
Today we'll focus on the non-trace mode. It tries to measure the path MTU with a bisection method varying the packet sizes between minimum MTU (92) and maximum MTU (specified with the -m parameter, default is 10000 bytes). The payload size of the first packet (without the -m flag) is thus 5046 bytes ((10000 + 92)/2).
On each iteration, the algorithm prints a “cryptic” sign indicating whether the ping with the current payload size succeeded. The following indications are given:
- '+': ICMP echo reply arrived
- '-': The ping failed (for various reasons, including exceeding the path MTU)
- 'u': ICMP destination unreachable response arrived, indicating blackhole or access-list.
- '.': timeout. The ping will be retried up to three times (or the number specified with the -r option).
ICMP unreachable is considered a successful response; at least we're measuring the path MTU up to the failure point
$ mturoute 10.0.3.3
* ICMP Fragmentation is not permitted. *
* Maximum payload is 10000 bytes. *
- ICMP payload of 5046 bytes failed..
- ICMP payload of 2569 bytes failed..
+ ICMP payload of 1330 bytes succeeded.
- ICMP payload of 1949 bytes failed..
- ICMP payload of 1639 bytes failed..
- ICMP payload of 1484 bytes failed..
+ ICMP payload of 1407 bytes succeeded.
- ICMP payload of 1445 bytes failed..
+ ICMP payload of 1426 bytes succeeded.
+ ICMP payload of 1435 bytes succeeded.
+ ICMP payload of 1440 bytes succeeded.
+ ICMP payload of 1442 bytes succeeded.
+ ICMP payload of 1443 bytes succeeded.
+ ICMP payload of 1444 bytes succeeded.
+ ICMP payload of 1444 bytes succeeded.
Path MTU: 1472 bytes.
Note: To use the debug-enabled version of mturoute, or the version that does not need VC++ runtime, download the new ZIP archive from my web site.
Show IP access lists attached to an interface
Log the NTP events
I almost started writing an EEM applet that would detect and log the changes in router’s system time caused by NTP synchronizations, but then I’ve decided to check the IOS documentation first and found the ntp logging command.
OSPF Router-Id Does Not Change When the Interface IP Address Changes
The venerable rules used to establish OSPF router ID on Cisco IOS are all over the Internet:
- Take the highest IP address of all loopback interfaces configured on the router when the OSPF process is started.
- If there is no loopback interface, take the highest IP address of an operating interface.
In the old days, when Cisco believed that the router ID had to match an interface address, this also implied that the router ID would have changed if the interface IP address changed (and we told the students that you have to use loopback interfaces to make your network stable, as the OSPF process would restart if the interface giving the router ID went down).