Category: IP routing

The IP-over-IP (usually GRE) tunnels (commonly in combination with IPSec to provide security) are frequently used when you want to transport private IP traffic over public IP network that does not support layer 3 VPNs. If you use the GRE tunnels in combination with default routing (or route summarization), you can get serious routing issues when the tunnel destination disappears, but a default (or summary) route in the IP routing table still covers it. You could work around this issue by deploying a routing protocol over the GRE tunnel (which could lead to hard to diagnose routing loops if you're not careful) or by using GRE keepalives introduced in IOS release 12.2(8)T.

The implementation of the GRE keepalives is amazing: the router sending the keepalive packet constructs a GRE packet that would be sent from the remote end back to itself (effectively building a GRE reply), sets the GRE protocol type to zero (to indicate the keepalive packet) and sends the whole packet through the tunnel (effectively encapsulating GRE reply into another GRE envelope). The receiving router strips the GRE envelope and routes the inside packet … which is the properly formatted GRE keepalive reply.

This trick allows you to implement different GRE keepalive timers on each end of the link. For example, the remote site might use fast keepalive timers to detect loss of primary link and switch over to a backup link, while the central site would use less frequent keepalive tests to detect failed remote site (if there is a single path to the remote site, you don't care too much when you detect it's down).

Every ingenious solution has its drawbacks and this one is no exception: if the receiving router protects its IP addresses (to stop spoofing attacks), it will drop the incoming GRE keepalive packet. Furthermore, a document available on Cisco's web describes the issues of using GRE keepalives in IPSec environment.

When I started exploring the details of MTU handling in Cisco IOS, I quickly got tired of analyzing various long printouts to extract the MTU sizes, so I wrote a Tcl script that display hardware, IP and MPLS MTUs in a tabular format. To install it on your router:

1. Download it from my web site and copy it to your router's flash or NVRAM.
2. Define an alias, for example alias exec mtu tclsh flash:displayMTU.tcl.

The script recognizes two parameters: the ip parameter displays only the interfaces that have IP configured and the mpls parameter displays only the MPLS-enabled interfaces.

An IOS device configured for IP+MPLS routing uses three different Maximum Transmission Unit (MTU) values:

• The hardware MTU configured with the mtu interface configuration command
• The IP MTU configured with the ip mtu interface configuration command
• The MPLS MTU configured with the mpls mtu interface configuration command

The hardware MTU specifies the maximum packet length the interface can support … or at least that's the theory behind it. In reality, longer packets can be sent (assuming the hardware interface chipset doesn't complain); therefore you can configure MPLS MTU to be larger than the interface MTU and still have a working network. Oversized packets might not be received correctly if the interface uses fixed-length buffers; platforms with scatter/gather architecture (also called particle buffers) usually survive incoming oversized packets.

IP MTU is used to determine whether a non-labeled IP packet forwarded through an interface has to be fragmented (the IP MTU has no impact on labeled IP packets). It has to be lower or equal to hardware MTU (and this limitation is enforced). If it equals the HW MTU, its value does not appear in the running configuration and it tracks the changes in HW MTU. For example, if you configure ip mtu 1300 on a Serial interface, it will appear in the running configuration as long as the hardware MTU is not equal to 1300 (and will not change as the HW MTU changes). However, as soon as the mtu 1300 is configured, the ip mtu 1300 command disappears from the configuration and the IP MTU yet again tracks the HW MTU.

The MPLS MTU determines the maximum size of a labeled IP packet (MPLS shim header + IP payload size). If the overall length of the labeled packet (including the shim header) is greater than the MPLS MTU, the packet is fragmented. The MPLS MTU can be greater than the HW MTU assuming the hardware architecture and interface chipset support that (and the router will warn you that you might be getting into trouble). Similar to the ip mtu command, the mpls mtu command will only appear in the running configuration if the MPLS MTU is different from the HW MTU. However, contrary to the behavior of the IP MTU, any change in HW MTU with the mtu configuration command also resets the MPLS MTU to HW MTU.

The behavior as described above was tested on a 3725 router running IOS release 12.4(15)T1. Although the MPLS MTU Command Changes document claims that you cannot set MPLS MTU larger than then interface MTU from IOS release 12.4(11)T, I was still able to do it in 12.4(15)T1.

Continuing from the previous mturoute-related post, this is how the mturoute utility behaves when you start it in traceroute mode (with the -t flag):

• Similar to Windows tracert, it tries to find the successive hops in the path by sending ICMP echo packets with increasing values of TTL field.

Contrary to Cisco IOS and most Unix systems that send UDP packets to high-numbered ports, tracert uses ICMP echo packets.

• For each router found in the path (= source IP address in the ICMP TTL exceeded message), mturoute tries to find path MTU to that hop using the same algorithm as in the ping mode.
• During the bisecting phase, the mturoute does not print all the messages it prints in the ping mode, but just the cryptic signs (+/-/u/.) indicating its progress. Their meaning is documented in the previous post.
• After the path MTU to the router under investigation is measured, mturoute reports the router's IP address and path MTU.

Yesterday's post has generated quite a few comments (obviously a tool like this comes handy :); some of you were unable to run the .exe file I've provided, others wondered about the unexpected results. While testing the first issue, I've figured out that:

• Any C program compiled with the free Visual C++ compiler from Microsoft requires runtime library that has to be installed separately. Update: not completely true, if you use change the runtime library to the non-DLL version (Project properties/C++/Code generation tab), the exe size increases, but the external dependencies are removed.
• The Visual C++ 2008 that I've used has no publicly available runtime library that you could install.

So I had to scrap my VC++ 2008 installation, download VC++ 2005, reinstall the Microsoft Platforms SDK and (after a few hours) recompiled the program:

• The new build can be downloaded from the same location.
• It includes a README file that documents the changes made to the source.
• To run it, you have to install the VC++ 2005 runtime library from Microsoft

. Update (2007-10-03): I've rebuilt the image with static runtime library, so the VC++ runtime DLL is no longer needed. Thanks to Vladimir Kocjancic for figuring this out for me.

After these changes, the utility should be able to execute on Vista as well.

Apart from the rebuild, I've fixed the ICMP destination network unreachable handling, which is considered identical to successful ping in the MTU measurement code (I still need to fix its handling in the trace part of the code).

There are also a few caveats when using this program on a Windows platform enabled for Path MTU discovery (default for the last few years):

• Whenever the Windows TCP stack receives an ICMP specifying the maximum MTU, it caches the reported MTU size (makes sense).
• The cached MTU sizes eventually expire (but I was not able to find any documentation on the expiration time).
• I was also not able to find any documented way of purging the path MTU cache. The command that works for me is the route -f which flushes the IP routing table.
• Obviously, after executing route -f, the DHCP-installed default route is gone, so you have to execute ipconfig /renew.

Note: Any hints on the internal workings of path MTU cache on Windows platforms are highly appreciated

I wanted to get in-depth details on how various MTU parameters interact in GRE/IPSec/MPLS environment. Before going into router configuration details, I wanted to have a tool that would reliably measure actual path MTU between the endpoints. After a while, Google gave me a usable link: supposedly the tracepath program on Linux does what I needed. As I'm a purely Windows user (for me, PCs are just a tool), I needed a Windows equivalent … and found mturoute, the utility that does exactly what I was looking for.

The MPLS VPN implementation on Cisco IOS has always allowed you to create VRF static routes that pointed to interfaces belonging to other VRFs. The feature can be used to implement interesting overlapping VPN (or common services VPN) designs, some of which are explained in the MPLS and VPN Architectures books.

However, quite often the ability to create inter-VRF static routes is considered a major security problem, as an operator configuration error could establish undesired inter-VPN connectivity. In these cases, use the no ip route static inter-vrf configuration command to prevent such routes from being installed in the VRF routing table.

You might also want to read a good explanation of MPLS VPN route leaking from Cisco systems

The Static routing with Catalyst 3750 post has generated a lot of good, creative ideas. Some of the proposed solutions were better than the others and some were simply not implementable (but nonetheless, had great creative potential :). Here is my list of the favorites:

A routing protocol: as a few of you have rightly pointed out, this is the best choice.

Aggressive Unidirectional Link Detection (UDLD): this is my second favorite, as it's a reliable link-level mechanism that will detect a break in the fiber cable … exactly the right tool for the job.

In a previous post, I've described how the track interface ip routing command reports incorrect interface state if you use IP Event Dampening feature. To track the actual IP routing readiness of an interface, you could use the following workaround:

• Create a static IP route pointing to the interface you want to test. Make sure this route is not redistributed into any routing protocols.
• Track the reachability of the static route

Here's an interesting scenario:

We have two sites, each using a Catalyst 3750 switch, and routing between them using static routes. There's a primary fiber link between them and we're using twisted-pair-to-fiber converters due to port limitations on Cat3750. These converters do not report fiber link down status correctly (the carrier is still present on twisted pair even if fiber is down), so the primary Ethernet interfaces do not go down if the fiber link breaks and the primary static route is not removed, requiring manual action to switch over to the backup link.

The setup is summarized in this diagram:

The track number interface name ip routing command is supposed to track an interface readiness to forward IP packets. In reality, it only tracks the interface line protocol status plus the IPCP status in case of PPP interfaces (as well as the actual presence of an IP address on the interface). If you configure IP Event Dampening (with the dampening) command, the interface might be suppressed (unavailable for IP routing), but the track object will report it as available (tested on IOS release 12.4(6)T). This could result in suboptimal HSRP/GLBP decisions if you use track objects to influence HSRP/GLBP priority or actual loss of data if you use such a track object to control policy-based routing.

One of my readers recently asked an interesting question: “How do you install a static route when an IP address is not reachable?”

Without going into the design reasons that prompted the question, you can actually track when IP SLA measurement fails with an obscure configuration syntax of the track objects that tracks when another track object fails.

One of my readers asked for a working configuration of the conditional OSPF default route advertisement feature. In my scenario, the OSPF default route would be announced whenever an Internet prefix (172.18.0.0/16) would be present in the IP routing table.

After I've published the Tcl script that displays the interface IP parameters in a formatted table, cos quickly pointed out a bug: I've expected the IP addresses in the address mask format. In the meantime, I've figured out the root cause of the problem (our remote labs are set to display IP masks in decimal format for compatibility reasons) and fixed the Tcl script. It temporarily sets the terminal ip netmask-format to bit-count before executing the show command. The new script recognizes three parameters:

• active: display only interfaces that are up/up;


• configured: display only interfaces with configured IP addresses (unnumbered interfaces using IP address of an interface without one count as configured since IOS reports their IP address as 0.0.0.0).


• address: displays IP address of the unnumbered interface, not the interface that it's borrowing the address from.

You can view the Tcl source or download it from my web site.

Here’s an interesting OSPF-related question I got::

“Which one is better: default-information originate or default-information originate always?”

As always, the answer is it depends. If your OSPF edge routers have external default routes (for example, static default routes toward the Internet, see the next diagram), you'd want them to announce the default route only when they have a default themselves (otherwise, they would attract the traffic and then blackhole it). In this case, you’d use default-information originate.