Category:  IP routing

I'm often promoting the idea of separating customer routing from core routing in the design articles I write. The only viable solution (unless you want to implement MPLS VPN and migrate customer routing into VPNv4) is to carry customer routes in BGP, redistributing them into BGP from other routing sources. On the other hand, I’m telling you that you should advertise only static IP prefixes into the public Internet. Obviously there’s a seeming disconnect between the two advices.

However, the dilemma is easily solved with the no-export BGP community that prevents an IP prefix from being advertised over EBGP sessions. Whenever you redistribute customer routes into BGP, you should attach the no-export community to them, ensuring that only the statically advertised IP prefixes will be propagated outside of your AS boundaries.

Brian Dennis wrote a long post about the unexpected side effects of the ip default-network command. The Cisco documentation describes the “side effects” but in an even more obscure manner.

What's really happening is this:

• If the parameter of the ip default-network command is a major network, it specifies the default route (how it gets inserted into the routing protocol you're using is a completely different story).
• If the parameter is a subnet of a major network, it specifies the default subnet for the network.

In any case, it's an obscure leftover from the classful days that should probably never be used today outside of a CCIE lab.

Olivier Guillemain has asked an interesting question: “how could I originate a default route into OSPF based on IP SLA (for example, based on pinging a remote IP address)?”

This is very easy to do when the router originating the default route into OSPF needs an SLA-based default route itself:

1. Configure IP SLA and a corresponding track object;
2. Configure a default route using reliable static routing
3. Advertise the default route into OSPF with the default-information originate router configuration command

The solution is a bit more complex when the router originating the default route into OSPF should not have a default route. In this case, you could use a routing trick:

1. Configure IP SLA and a corresponding track object as before;
2. Use reliable static routing to configure a static host route for a bogus IP address (for example, 10.0.0.1/32) pointing to null0 (for example, ip route 10.0.0.1 255.255.255.255 null 0 track 100). Obviously, this host route should not be redistributed into any routing protocol.
3. Conditionally advertise default route into OSPF based on presence of the static host route.

In the last few months I ran across a number of IP fragmentation issues. Unfortunately I also encountered a lot of misconceptions about IP fragmentation, its impact on GRE and IPSec, as well as the fragmentation-related mechanisms like MTU Path Discovery. I documented most of what I found in the The Never-Ending Story of IP Fragmentation.

You've probably already heard the phrase "When the only tool you have is a hammer, everything looks like a nail" (and seen people acting according to it). Likewise, if you have an IOS release with EEM support, a lot of things that would require smart design could be solved in a brute-force way with a few EEM applets. For example, the problem of the BGP default route could be solved “easily” with a few applets that track syslog messages reporting when the BGP neighbors go up/down.

Speaking of Joe Harris … he just wrote a great post about the IP routing table profiling available in Cisco IOS and the ways it can be used to monitor the stability of your network.

The IP-over-IP (usually GRE) tunnels (commonly in combination with IPSec to provide security) are frequently used when you want to transport private IP traffic over public IP network that does not support layer 3 VPNs. If you use the GRE tunnels in combination with default routing (or route summarization), you can get serious routing issues when the tunnel destination disappears, but a default (or summary) route in the IP routing table still covers it. You could work around this issue by deploying a routing protocol over the GRE tunnel (which could lead to hard to diagnose routing loops if you're not careful) or by using GRE keepalives introduced in IOS release 12.2(8)T.

The implementation of the GRE keepalives is amazing: the router sending the keepalive packet constructs a GRE packet that would be sent from the remote end back to itself (effectively building a GRE reply), sets the GRE protocol type to zero (to indicate the keepalive packet) and sends the whole packet through the tunnel (effectively encapsulating GRE reply into another GRE envelope). The receiving router strips the GRE envelope and routes the inside packet … which is the properly formatted GRE keepalive reply.

This trick allows you to implement different GRE keepalive timers on each end of the link. For example, the remote site might use fast keepalive timers to detect loss of primary link and switch over to a backup link, while the central site would use less frequent keepalive tests to detect failed remote site (if there is a single path to the remote site, you don't care too much when you detect it's down).

Every ingenious solution has its drawbacks and this one is no exception: if the receiving router protects its IP addresses (to stop spoofing attacks), it will drop the incoming GRE keepalive packet. Furthermore, a document available on Cisco's web describes the issues of using GRE keepalives in IPSec environment.

When I started exploring the details of MTU handling in Cisco IOS, I quickly got tired of analyzing various long printouts to extract the MTU sizes, so I wrote a Tcl script that display hardware, IP and MPLS MTUs in a tabular format. To install it on your router:

1. Download it from my web site and copy it to your router's flash or NVRAM.
2. Define an alias, for example alias exec mtu tclsh flash:displayMTU.tcl.

The script recognizes two parameters: the ip parameter displays only the interfaces that have IP configured and the mpls parameter displays only the MPLS-enabled interfaces.

An IOS device configured for IP+MPLS routing uses three different Maximum Transmission Unit (MTU) values:

• The hardware MTU configured with the mtu interface configuration command
• The IP MTU configured with the ip mtu interface configuration command
• The MPLS MTU configured with the mpls mtu interface configuration command

The hardware MTU specifies the maximum packet length the interface can support … or at least that's the theory behind it. In reality, longer packets can be sent (assuming the hardware interface chipset doesn't complain); therefore you can configure MPLS MTU to be larger than the interface MTU and still have a working network. Oversized packets might not be received correctly if the interface uses fixed-length buffers; platforms with scatter/gather architecture (also called particle buffers) usually survive incoming oversized packets.

IP MTU is used to determine whether a non-labeled IP packet forwarded through an interface has to be fragmented (the IP MTU has no impact on labeled IP packets). It has to be lower or equal to hardware MTU (and this limitation is enforced). If it equals the HW MTU, its value does not appear in the running configuration and it tracks the changes in HW MTU. For example, if you configure ip mtu 1300 on a Serial interface, it will appear in the running configuration as long as the hardware MTU is not equal to 1300 (and will not change as the HW MTU changes). However, as soon as the mtu 1300 is configured, the ip mtu 1300 command disappears from the configuration and the IP MTU yet again tracks the HW MTU.

The MPLS MTU determines the maximum size of a labeled IP packet (MPLS shim header + IP payload size). If the overall length of the labeled packet (including the shim header) is greater than the MPLS MTU, the packet is fragmented. The MPLS MTU can be greater than the HW MTU assuming the hardware architecture and interface chipset support that (and the router will warn you that you might be getting into trouble). Similar to the ip mtu command, the mpls mtu command will only appear in the running configuration if the MPLS MTU is different from the HW MTU. However, contrary to the behavior of the IP MTU, any change in HW MTU with the mtu configuration command also resets the MPLS MTU to HW MTU.

The behavior as described above was tested on a 3725 router running IOS release 12.4(15)T1. Although the MPLS MTU Command Changes document claims that you cannot set MPLS MTU larger than then interface MTU from IOS release 12.4(11)T, I was still able to do it in 12.4(15)T1.

Continuing from the previous mturoute-related post, this is how the mturoute utility behaves when you start it in traceroute mode (with the -t flag):

• Similar to Windows tracert, it tries to find the successive hops in the path by sending ICMP echo packets with increasing values of TTL field.

Contrary to Cisco IOS and most Unix systems that send UDP packets to high-numbered ports, tracert uses ICMP echo packets.

• For each router found in the path (= source IP address in the ICMP TTL exceeded message), mturoute tries to find path MTU to that hop using the same algorithm as in the ping mode.
• During the bisecting phase, the mturoute does not print all the messages it prints in the ping mode, but just the cryptic signs (+/-/u/.) indicating its progress. Their meaning is documented in the previous post.
• After the path MTU to the router under investigation is measured, mturoute reports the router's IP address and path MTU.

Yesterday's post has generated quite a few comments (obviously a tool like this comes handy :); some of you were unable to run the .exe file I've provided, others wondered about the unexpected results. While testing the first issue, I've figured out that:

• Any C program compiled with the free Visual C++ compiler from Microsoft requires runtime library that has to be installed separately. Update: not completely true, if you use change the runtime library to the non-DLL version (Project properties/C++/Code generation tab), the exe size increases, but the external dependencies are removed.
• The Visual C++ 2008 that I've used has no publicly available runtime library that you could install.

So I had to scrap my VC++ 2008 installation, download VC++ 2005, reinstall the Microsoft Platforms SDK and (after a few hours) recompiled the program:

• The new build can be downloaded from the same location.
• It includes a README file that documents the changes made to the source.
• To run it, you have to install the VC++ 2005 runtime library from Microsoft

. Update (2007-10-03): I've rebuilt the image with static runtime library, so the VC++ runtime DLL is no longer needed. Thanks to Vladimir Kocjancic for figuring this out for me.

After these changes, the utility should be able to execute on Vista as well.

Apart from the rebuild, I've fixed the ICMP destination network unreachable handling, which is considered identical to successful ping in the MTU measurement code (I still need to fix its handling in the trace part of the code).

There are also a few caveats when using this program on a Windows platform enabled for Path MTU discovery (default for the last few years):

• Whenever the Windows TCP stack receives an ICMP specifying the maximum MTU, it caches the reported MTU size (makes sense).
• The cached MTU sizes eventually expire (but I was not able to find any documentation on the expiration time).
• I was also not able to find any documented way of purging the path MTU cache. The command that works for me is the route -f which flushes the IP routing table.
• Obviously, after executing route -f, the DHCP-installed default route is gone, so you have to execute ipconfig /renew.

Note: Any hints on the internal workings of path MTU cache on Windows platforms are highly appreciated

I wanted to get in-depth details on how various MTU parameters interact in GRE/IPSec/MPLS environment. Before going into router configuration details, I wanted to have a tool that would reliably measure actual path MTU between the endpoints. After a while, Google gave me a usable link: supposedly the tracepath program on Linux does what I needed. As I'm a purely Windows user (for me, PCs are just a tool), I needed a Windows equivalent … and found mturoute, the utility that does exactly what I was looking for.

The MPLS VPN implementation on Cisco IOS has always allowed you to create VRF static routes that pointed to interfaces belonging to other VRFs. The feature can be used to implement interesting overlapping VPN (or common services VPN) designs, some of which are explained in the MPLS and VPN Architectures books.

However, quite often the ability to create inter-VRF static routes is considered a major security problem, as an operator configuration error could establish undesired inter-VPN connectivity. In these cases, use the no ip route static inter-vrf configuration command to prevent such routes from being installed in the VRF routing table.

You might also want to read a good explanation of MPLS VPN route leaking from Cisco systems

The Static routing with Catalyst 3750 post has generated a lot of good, creative ideas. Some of the proposed solutions were better than the others and some were simply not implementable (but nonetheless, had great creative potential :). Here is my list of the favorites:

A routing protocol: as a few of you have rightly pointed out, this is the best choice.

Aggressive Unidirectional Link Detection (UDLD): this is my second favorite, as it's a reliable link-level mechanism that will detect a break in the fiber cable … exactly the right tool for the job.

In a previous post, I've described how the track interface ip routing command reports incorrect interface state if you use IP Event Dampening feature. To track the actual IP routing readiness of an interface, you could use the following workaround:

• Create a static IP route pointing to the interface you want to test. Make sure this route is not redistributed into any routing protocols.
• Track the reachability of the static route