Speaking of Joe Harris … he just wrote a great post about the IP routing table profiling available in Cisco IOS and the ways it can be used to monitor the stability of your network.

Joe Harris published an excellent post detailing how you can use Flexible Packet Matching to recognize (and potentially block) Skype traffic. The solution depends on recognizing the first four bytes sent by the Skype application in a TCP session. While this is a great idea, you have to be aware that there's always a non-zero chance of false positives, more so as the described filter is testing the beginning of the payload in every TCP packet (not just the first data packet in the session).

My recent post about problems with old modems has generated a lot of comments with some very useful ideas, but nobody addressed the question “why was a long string of ones not a problem?”, so let's start there. Almost all WAN synchronous protocols in use today are descendants of venerable SDLC invented by IBM more than 30 years ago. SDLC was later extended to support connectionless and balanced modes, resulting in HDLC. PPP is just an extension of HDLC, adding support for negotiations and standard layer-3 protocol demultiplexing. In SDLC, IBM also solved the frame delimiting and associated escape character problem inherent in previous protocols like BSC (DLE was used in BSC) by introducing bit stuffing: a zero would be inserted after five consecutive ones (and silently removed by the receiver) to differentiate the regular data stream from framing (six consecutive ones) and abort (more than six consecutive ones) sequences. Thus, the HDLC (or PPP) data stream can never contain more than six consecutive ones and the long sequences of ones never cause synchronization loss.

IBM obviously also had problems with bad modems and solved it with the NRZI encoding that was part of SDLC standard (and a major pain in the good old days when the appliques on the old Cisco routers did not support it and we've been trying hard to penetrate IBM accounts). You can still configure NRZI encoding on most routers' serial links (it might depend on the actual hardware platform) with the nrzi-encoding interface configuration command (you had to do it with jumpers in the AGS+). Incidentally, changing interface encoding to NRZI was really helpful when you had to break things in the preparation for the troubleshooting part of the original CCIE lab).

Enough theory, let's summarize the proposed solutions:

• The nrzi-encoding (if available) is the best one, as it reliably solves the problem, is transparent and does not incur additional overhead.
• Compression or encryption are OK, but they result in significant CPU overhead (unless you have hardware encryption/compression modules) and might (at least in theory) still produce a long sequence of zeroes, although with a very low probability. IPSec also introduces overhead due to additional IPSec headers.
• LFI (effectively multilink PPP over a single link) is also a good solution, as the PPP framing and MLPPP headers break the long sequences of zeroes (you might have to fine-tune the fragment size with ppp multilink fragment size configuration command), but it introduces overhead on the WAN link.
• IP fragmentation would work, but would be quite bandwidth-consuming. If the fragmentation would be performed by the router, the overhead would be 20 bytes per fragment (IP header), if the sending host performs the fragmentation, the overhead is 40 bytes per fragment for TCP sessions. For example, if we reduce the IP MTU size to 256 bytes, the TCP session overhead is over 18% (and we were scoffing at the ATM designers that made us live with 10% overhead).

There were also a few suggestions that would not work very well:

• The invert data command would only help if the modem has problems with long strings of zeroes, not with long strings of the same value.
• The tunnel key command just sets a 4-byte field in the GRE header but does not affect the encapsulated data at all.

William Chu sent me a working configuration he uses to measure jitter with the IP SLA tool and react to excessive jitter on the primary link. First you have to create the jitter probe with the IP SLA commands:

ip sla monitor 3000
 type jitter →
   dest-ipaddr 199.11.18.168 dest-port 12333 →
   source-ipaddr 199.11.18.169 codec g729a →
   codec-numpackets 100
 tos 184
 frequency 10

Note: The continuation character (→) indicates that the configuration command spans multiple lines

Next you have to define the IP SLA reaction to excessive jitter. William configured his router to react when the jitter exceeds 300 milliseconds and returns back to normal when the jitter falls below 290 milliseconds (some hysteresis is always a good thing).

ip sla monitor reaction-configuration 3000 →
  react MOS threshold-value 300 290 →
  threshold-type consecutive →
  action-type trapOnly

As the last step in the SLA configuration, you have to start the probe:

ip sla monitor schedule 3000 →
  life forever start-time now

After the SLA probe and out-of-bounds reaction have been configured, the router will generate syslog messages whenever the jitter gets above the threshold as well as when it falls below the second threshold. You can then use the EEM applets to act on the syslog messages:

event manager applet MOS-Below
 event syslog occurs 1 period 120 →
   pattern "Threshold below for MOS"
 ... actions ...
!
event manager applet MOS-Above
 event syslog occurs 1 period 120 →
   pattern "Threshold exceeded for MOS"
 ... actions ...

In the pre-DSL days, you had two options to get a short-haul high-speed link (at least in Europe): take E1 (or fractional E1) from a telecom (which was more expensive than a highway robbery, as the cost was recurring) or use baseband modems with proprietary encoding techniques on physical copper wires (assuming you could get them). As it turned out, some of these encoding techniques were not as good as the others (but the equipment was relatively cheap, so the budget limits usually forced the decision). We had our own share of modem-related problems, but they were never as bad as what I've heard from one of my students: his modems would lose synchronization when transmitting a long string of zeroes over a regular synchronous serial interface; ping ip 1.2.3.4 size 1000 data 0000 would be enough to bring down the link.

And now two questions for you:

• What could you do on the router to fix this problem?
• Why was the synchronization retained when transmitting a long string of ones?

Did you believe MPLS TE was a quality-of-service feature? Did someone persuade you it’s mandatory to run OSPF or IS-IS if you want to deploy MPLS TE? I’ve collected a few more myths like these two and explained the actual facts behind them in 10 MPLS traffic engineering myths and half truths published by ComputerWeekly..

In a previous post, I've described how you can turn your router into a TFTP server. As you can configure the router to serve any file residing on it, you can also pull startup and running configuration from it with TFTP, providing that you configure:

tftp-server nvram:startup-config
tftp-server system:running-config

Warning: Due to total lack of any security features in TFTP protocol, use this functionality only in lab environment.

I should probably write this one on April 1st, but maybe October 31st is not such a bad choice after all … if you configure no service prompt config, the configuration prompt is gone; when you enter the configuration mode with the configure terminal command, you get an empty line (like you did with Cisco software release 9.1 some 15 years ago). Similarly, you can disable command-line editing with the no editing line configuration command or terminal no editing exec-level command. If only there would be a way to disable the context-sensitive help :)

I did a few follow-up tests with the distribute-list in OSPF configuration command and stumbled across a few interesting facts (IOS release 12.4(15)T1 on a 3725 platform):

• Although the router allows you to configure distribute-list acl in interface, it does not work. Routes received through that interface (or having the interface as the next-hop) are not filtered.
• When you apply the distribute-list in command, the routing table is not changed. Clearing the IP routing table does not help, you have to clear ALL OSPF processes (including bringing down all OSPF adjacencies) with the clear ip ospf process command for the route filter to take effect.
• The same limitations don't apply in the other direction: when you remove the distribute-list in, SPF is triggered and the routes appear in the IP routing table automatically.
• The somewhat undocumented gateway option of the distribute-list in command works, but not quite as I would expect: the IP next hop, not the router-ID of the router advertising the IP prefix is matched by the prefix-list.

And, last but not least, I've lab-verified my previous claim: applying the distribute-list in on a transit router can result in a black hole, as the LSAs themselves are not filtered.

John S. Pumphrey recently asked an interesting question: “Can the router send an e-mail when an interface goes down?” The enterprisey solution is obvious: deploy a high-end EMS to collect SNMP traps and use its API to write a custom module that would use a MQ interface to alert the operator. Fortunately, Event Manager applets in Cisco IOS provide action mail command (available in 12.3(14)T and 12.4) that can send an e-mail to a SMTP server straight from the router.

There are two ways you can detect that an interface went down with EEM: either you track the interface status with a track object and start an EEM applet when the track object changes state or you catch the syslog messages reporting that the interface line protocol changed state to down. The second approach is obviously more generic, as a single applet can act on multiple interfaces.

event manager applet MailOnIfDown
 event syslog occurs 1 →
    pattern "LINEPROTO-5-UPDOWN.*to down" →
    period 1

The action mail command specifies the mail server's address (use a hostname and DNS lookup or ip host configuration command to make the EEM applet more generic), from and to address, message subject and body. In each of these fields, you can use EEM environment variables that you can define with the event manager environment configuration command. Each EEM event also defines a few environment variables that you can use (see the table of EEM system-defined variables on CCO). For example, you can define the e-mail recipient in the router's configuration and use the _syslog_msg variable to include the syslog message in the e-mail body:

event manager environment _ifDown_rcpt [email protected]
!
event manager applet MailOnIfDown
 event syslog occurs 1 →
    pattern "LINEPROTO-5-UPDOWN.*to down" →
    period 1
 action 1.0 mail server "mail-gw" →
    to "$_ifDown_rcpt" from "[email protected]" →
    subject "Interface down on R1" →
    body "$_syslog_msg"

You can make the applet even more generic with the help of action info type routername command, which stores the current router's name into the $_info_routername environment variable:

event manager environment _ifDown_rcpt [email protected]
!
event manager applet MailOnIfDown
 event syslog occurs 1 →
    pattern "LINEPROTO-5-UPDOWN.*to down" →
    period 1
 action 1.0 info type routername
 action 2.0 mail server "mail-gw" →
    to "$_ifDown_rcpt

" from "$_info_routername@lab.com" →
    subject "Interface down on $_info_routername" →
    body "$_syslog_msg"

Note: This article is part of You've asked for it series.

A while ago I wrote about cached CEF adjacencies and the impact they have on ARP caching. If you ever need to, you can debug them with the debug ip cef table command. As this command might produce a lot of output in a production network, always use it in combination with an access-list that limits the debugging to the selected address range.

Alternatively, you can use the debug arp adjacency command, but you cannot limit its output with an access-list

This problem is rare but tantalizing enough to warrant mentioning: OSPF neighbors are forever stuck in the EXSTART state (occasionally going DOWN and back to EXSTART).

The IP-over-IP (usually GRE) tunnels (commonly in combination with IPSec to provide security) are frequently used when you want to transport private IP traffic over public IP network that does not support layer 3 VPNs. If you use the GRE tunnels in combination with default routing (or route summarization), you can get serious routing issues when the tunnel destination disappears, but a default (or summary) route in the IP routing table still covers it. You could work around this issue by deploying a routing protocol over the GRE tunnel (which could lead to hard to diagnose routing loops if you're not careful) or by using GRE keepalives introduced in IOS release 12.2(8)T.

The implementation of the GRE keepalives is amazing: the router sending the keepalive packet constructs a GRE packet that would be sent from the remote end back to itself (effectively building a GRE reply), sets the GRE protocol type to zero (to indicate the keepalive packet) and sends the whole packet through the tunnel (effectively encapsulating GRE reply into another GRE envelope). The receiving router strips the GRE envelope and routes the inside packet … which is the properly formatted GRE keepalive reply.

This trick allows you to implement different GRE keepalive timers on each end of the link. For example, the remote site might use fast keepalive timers to detect loss of primary link and switch over to a backup link, while the central site would use less frequent keepalive tests to detect failed remote site (if there is a single path to the remote site, you don't care too much when you detect it's down).

Every ingenious solution has its drawbacks and this one is no exception: if the receiving router protects its IP addresses (to stop spoofing attacks), it will drop the incoming GRE keepalive packet. Furthermore, a document available on Cisco's web describes the issues of using GRE keepalives in IPSec environment.

When I started exploring the details of MTU handling in Cisco IOS, I quickly got tired of analyzing various long printouts to extract the MTU sizes, so I wrote a Tcl script that display hardware, IP and MPLS MTUs in a tabular format. To install it on your router:

1. Download it from my web site and copy it to your router's flash or NVRAM.
2. Define an alias, for example alias exec mtu tclsh flash:displayMTU.tcl.

The script recognizes two parameters: the ip parameter displays only the interfaces that have IP configured and the mpls parameter displays only the MPLS-enabled interfaces.

An IOS device configured for IP+MPLS routing uses three different Maximum Transmission Unit (MTU) values:

• The hardware MTU configured with the mtu interface configuration command
• The IP MTU configured with the ip mtu interface configuration command
• The MPLS MTU configured with the mpls mtu interface configuration command

The hardware MTU specifies the maximum packet length the interface can support … or at least that's the theory behind it. In reality, longer packets can be sent (assuming the hardware interface chipset doesn't complain); therefore you can configure MPLS MTU to be larger than the interface MTU and still have a working network. Oversized packets might not be received correctly if the interface uses fixed-length buffers; platforms with scatter/gather architecture (also called particle buffers) usually survive incoming oversized packets.

IP MTU is used to determine whether a non-labeled IP packet forwarded through an interface has to be fragmented (the IP MTU has no impact on labeled IP packets). It has to be lower or equal to hardware MTU (and this limitation is enforced). If it equals the HW MTU, its value does not appear in the running configuration and it tracks the changes in HW MTU. For example, if you configure ip mtu 1300 on a Serial interface, it will appear in the running configuration as long as the hardware MTU is not equal to 1300 (and will not change as the HW MTU changes). However, as soon as the mtu 1300 is configured, the ip mtu 1300 command disappears from the configuration and the IP MTU yet again tracks the HW MTU.

The MPLS MTU determines the maximum size of a labeled IP packet (MPLS shim header + IP payload size). If the overall length of the labeled packet (including the shim header) is greater than the MPLS MTU, the packet is fragmented. The MPLS MTU can be greater than the HW MTU assuming the hardware architecture and interface chipset support that (and the router will warn you that you might be getting into trouble). Similar to the ip mtu command, the mpls mtu command will only appear in the running configuration if the MPLS MTU is different from the HW MTU. However, contrary to the behavior of the IP MTU, any change in HW MTU with the mtu configuration command also resets the MPLS MTU to HW MTU.

The behavior as described above was tested on a 3725 router running IOS release 12.4(15)T1. Although the MPLS MTU Command Changes document claims that you cannot set MPLS MTU larger than then interface MTU from IOS release 12.4(11)T, I was still able to do it in 12.4(15)T1.