Your browser failed to load CSS style sheets. Your browser or web proxy might not support elliptic-curve TLS

Building network automation solutions

9 module online course

Start now!

Update: mturoute

Yesterday's post has generated quite a few comments (obviously a tool like this comes handy :); some of you were unable to run the .exe file I've provided, others wondered about the unexpected results. While testing the first issue, I've figured out that:
  • Any C program compiled with the free Visual C++ compiler from Microsoft requires runtime library that has to be installed separately. Update: not completely true, if you use change the runtime library to the non-DLL version (Project properties/C++/Code generation tab), the exe size increases, but the external dependencies are removed.
  • The Visual C++ 2008 that I've used has no publicly available runtime library that you could install.
So I had to scrap my VC++ 2008 installation, download VC++ 2005, reinstall the Microsoft Platforms SDK and (after a few hours) recompiled the program:
  • The new build can be downloaded from the same location.
  • It includes a README file that documents the changes made to the source.
  • To run it, you have to install the VC++ 2005 runtime library from Microsoft. Update (2007-10-03): I've rebuilt the image with static runtime library, so the VC++ runtime DLL is no longer needed. Thanks to Vladimir Kocjancic for figuring this out for me.
  • After these changes, the utility should be able to execute on Vista as well.
Apart from the rebuild, I've fixed the ICMP destination network unreachable handling, which is considered identical to successful ping in the MTU measurement code (I still need to fix its handling in the trace part of the code).

There are also a few caveats when using this program on a Windows platform enabled for Path MTU discovery (default for the last few years):
  • Whenever the Windows TCP stack receives an ICMP specifying the maximum MTU, it caches the reported MTU size (makes sense).
  • The cached MTU sizes eventually expire (but I was not able to find any documentation on the expiration time).
  • I was also not able to find any documented way of purging the path MTU cache. The command that works for me is the route -f which flushes the IP routing table.
  • Obviously, after executing route -f, the DHCP-installed default route is gone, so you have to execute ipconfig /renew.

Note: Any hints on the internal workings of path MTU cache on Windows platforms are highly appreciated

see 2 comments

mturoute: A utility that measures hop-by-hop path MTU

I wanted to get in-depth details on how various MTU parameters interact in GRE/IPSec/MPLS environment. Before going into router configuration details, I wanted to have a tool that would reliably measure actual path MTU between the endpoints. After a while, Google gave me a usable link: supposedly the tracepath program on Linux does what I needed. As I'm a purely Windows user (for me, PCs are just a tool), I needed a Windows equivalent … and found mturoute, the utility that does exactly what I was looking for.Unfortunately, the original mturoute had an interesting bug: ICMP unreachable generated due to DF bit on oversized packet was accepted as a successful ping. The second run of the program always reported the correct MTU size (as Windows caches the maximum MTU per destination host), but I wanted to be more precise. Half a day later, after installing Windows SDK and Visual Studio Express on my PC, rediscovering my C programming skills and reading a lot of Winsock documentation, I've managed to fix the bugs and even add a “retry on ping timeout” feature. You can download the fixed version (source + exe) from the Articles area of my web site. It was compiled and tested on Windows XP, if you can test it on other platforms (2000, Vista), please let me know the results.

Here is a sample run of the program (the reduced path MTU is due to an MPLS-enabled GRE tunnel in the path):
$ mturoute -t
mturoute to, 30 hops max, variable sized packets
* ICMP Fragmentation is not permitted. *
* Maximum payload is 10000 bytes. *
1 --+---+++-+++-++ host: max: 1500 bytes
2 --+---++---+++ host: max: 1476 bytes
3 --+---+-+++++++ host: max: 1472 bytes
see 11 comments

Securing Networks with Cisco Routers and Switches

We have just released the new version of the Securing Networks with Cisco Routers and Switches (SNRS) remote lab exercises. They are an ideal companion to books or e-learning material if you're preparing for the CCSP exam. You can also use them as a great practice environment if you have to support security-related IOS features in your network, but simply don't have the extra equipment to test them out before deploying them.

As a side note, what really amazes me is the fact that Cisco has rolled out a mainstream certification course that supports pretty recent features (up to IOS release 12.4(6)T), including control-plane policing, management-plane protection, zone-based firewalls and Web VPN.

More information is available here.

see 1 comments

Stop Inter-VRF static route leaking

The MPLS VPN implementation on Cisco IOS has always allowed you to create VRF static routes that pointed to interfaces belonging to other VRFs. The feature can be used to implement interesting overlapping VPN (or common services VPN) designs, some of which are explained in the MPLS and VPN Architectures books.

However, quite often the ability to create inter-VRF static routes is considered a major security problem, as an operator configuration error could establish undesired inter-VPN connectivity. In these cases, use the no ip route static inter-vrf configuration command to prevent such routes from being installed in the VRF routing table.

You might also want to read a good explanation of MPLS VPN route leaking from Cisco systems

Add comment

Router as a TFTP server

Shaun needed an extra TFTP server in CCNP labs and asked whether you could use a router to act as one. The read-only (download only) TFTP functionality has been available in Cisco IOS for a long time, but the common wisdom was that you could only use the TFTP server function to serve current IOS image.

Fortunately, as of IOS 11.0, the function is more generic; you can serve any file residing on the router (you still cannot upload files), but you have to declare each file to be served with the tftp-server path global configuration command. You could even specify an alias to have the file available under a different name and attach an access list to each configured file to restrict its availability.

Note: This article is part of You've asked for it series.

see 4 comments

RIP multicast updates are sent with TTL of 2

The “culprit” for today's post is Francois Ropert, who provides a nice summary of Cisco-related blogs in his Planet Cisco web site. Reading the CCIE-related posts, I've stumbled across a post by Ethan Banks wondering whether the RIP updates are really sent with IP TTL set to 2. He concluded that this information cannot be deduced from the debugging outputs (anyone having an idea how you can get this information on the router?) and that he would need a sniffer to figure it out … and I thought: What a nice job for Dynamips.

I took one of my standard lab topologies (three routers in a triangle running OSPF between them, see the figure below) and started it in Dynagen.

When the routers were up and running I've configured RIP on all three of them:
router rip
 version 2
Next I've used Dynagen to start the packet capture on the first serial port of the R1 with the capture R1 s1/0 rip.cap PPP command. After a minute, I've stopped the capture with the no capture R1 s1/0 command and opened the rip.cap file with Wireshark. The results are shown below (click to enlarge); RIP multicast updates are actually sent with TTL 2 (at least in IOS release 12.4(15)T).

I've used the same technique to figure out that the RIP unicast updates (configured with the neighbor address router configuration command) are sent with TTL 255.

see 9 comments

EEM syslog messages look like debugging messages

If you use EEM applets with action syslog commands, the messages generated by those commands will be displayed in the same format as the debugging messages, regardless of their severity. This might be a problem if you decide to use different timestamps for debugging and regular syslog messages (or disable debugging timestamps completely).For example, the following configuration ...
service timestamps debug uptime
service timestamps log datetime
event manager applet Serial_Down
 event track 1 state any
 action 1.0 syslog priority errors msg "IP routing on Serial 0/1/0 is $_track_state"
... produces these messages after an interface failure ...
Aug 14 11:18:32: %LINK-3-UPDOWN: Interface Serial0/1/0, changed state to up
1d16h: %HA_EM-3-LOG: Serial_Down: IP routing on Serial 0/1/0 is up
Add comment

Five routers on your laptop

In case you've missed yesterday's post … the weather was just way too good to stay in the office :) However, even if I would decide to work on my routers, I could take them with me (well, the laptop would be a bit heavy and the sun was too bright) thanks to Christophe Fillot (Dynamips) and Greg Anuzelli (Dynagen).

In case you haven't heard about Dynamips/Dynagen yet: Dynamips emulates a variety of IOS platforms (from 2600 to 7200) on Intel platform and Dynagen provides friendlier user interface (more than friendly enough for me, probably too cryptic for GUI addicts). I've seen Dynamips a year or two ago, checked what it can do and decided to stay with the real routers in a remote lab environment. In the meantime, the software has improved drastically, allowing you to test all sorts of IOS features and topologies, as long as you don't expect QoS to work or real-time features to act in real-time (simulation is, after all, a bit slower than the real life).

To start using this tool, download it from, read the tutorial and you're in business. I will also start providing more interesting scenarios in the dynagen configuration file format.

Let me conclude with a few tips:
  • If you don't need 7200-specific features, select 37xx or 26xx platform, it consumes less virtual memory per router.
  • Setting idlepc is mandatory if you want to have decent response. Read the tutorial, the idlepc section is great.
  • Unzip the IOS files. With uncompressed files, the routers are ready to be configured in under a minute on my laptop; if the IOS image is compressed, it takes several minutes.
  • If you have larger topologies, use GhostIOS and Sparsemem features.
  • Reduce the size of NVRAM and Flash to minimum that would work. These are stored as persistent files on your disk; you can have 256MB Flash if you want, but then you'll have 256MB less of your hard drive (per router).

With all the above-mentioned features enabled, I was easily running eight 3700-series routers on my laptop (IBM T60).

see 13 comments

Frame Relay congestion management

In the “good old days” we've been teaching our students that although a router can act as a Frame Relay switch, it supports only the rudimentary functionality of switching the packets, but not the policing/marking features available in Frame Relay switches. That hasn't been true for a while - in IOS release 12.1T, Cisco has introduced the congestion management features. You can specify the congestion management per-interface (with the frame-relay congestion-management interface configuration command) and set the DE drop/ECN mark percentages for all PVCs on the interface or you can set the parameters within a map-class.

I don't know how useful this feature is to you; I was fond of finding it because it solves some interesting problems I had a (long) while ago. If you need more in-depth description or actual configurations, post a comment or send me a message.

see 2 comments

Assigning server IP addresses with DHCP

Using DHCP to assign server IP addresses is usually not a wise decision. To start with, you have to define static DHCP mappings, which rely on client-id attribute in the DHCP request (usually the MAC address of the client). For me, the easiest way to find the correct client ID is as follows:

  • Use DHCP to assign the IP address to the server
  • Note the newly assigned IP address
  • Use the show ip dhcp bindings | include ip-address command to display the client-id to IP address binding.
  • Create a static DHCP mapping (for example, by configuring a host DHCP pool on the router) and release/renew IP address on the server

Of course, if the Ethernet adapter in the server is replaced, the static mapping stops working. The only reliable workaround I've found so far is to assign a locally-administered MAC address to the server's LAN adapter (if anyone has figured out a way to assign ASCII client-ID to a Windows server, let me know). To do it on Windows XP, use the Advanced properties in the adapter configuration window.

Remember: locally-administered MAC addresses on Ethernet networks start with 02xx.

After you've configured MAC address on the server, prepend 01 to it and insert a dot after every fourth character to get the client-ID you need to enter on the DHCP server. For example, the MAC address 0200.1000.1234 becomes client-id 0102.0010.0012.34, and the static DHCP pool on a router is configured as follows:

ip dhcp pool Server_static
client-identifier 0102.0010.0012.34
see 8 comments

CEF accounting

The last week's creative challenge was obviously too easy; a number of readers quickly realized that the CEF accounting can do what we need (and I have to admit I've completely missed it).

However, when I started to explore the various CEF accounting features, it turned out the whole thing is not as simple as it looks. To start with, the ip cef accounting global configuration command configures three completely unrelated accounting features: per-prefix accounting (that we need), traffic matrix accounting (configured with the non-recursive keyword) and prefix-length accounting.

The per-prefix accounting is the easiest one to understand: every time a packet is forwarded through a CEF lookup, the counters attached to the CEF prefix entry are increased. To clear the CEF counters, you can use the clear ip cef address prefix-statistics command. The per-prefix counters are also lost when the IP prefix is removed from the CEF table (for example, because it temporarily disappears from the IP routing table during network convergence process). The CEF per-prefix accounting is thus less reliable than other accounting mechanisms (for example, IP accounting).

Note: The CEF per-prefix counters are always present; if the CEF per-prefix accounting is not configured, they simply remain zero.

Last but not least, you don't need the detail keyword if you want to display the CEF accounting data for a particular prefix. The show ip cef address mask command is enough. And, finally, if you're running IOS release 12.2SB or 12.2XN, you can inspect the CEF counters with SNMP.

see 2 comments

Display OSPF neighbor sorted by OSPF process ID

I had two issues with the show ip ospf neighbors command:
  • It is not sorted by the OSPF process ID, so you get a mess if you have more than one OSPF process and don't specify the process ID in the show command
  • It does not display the OSPF area the neighbor belongs to
To fix both problems, I wrote a Tcl script that displays OSPF neighbors sorted by process ID and includes the fields I wanted to have. To install it on your router:
  1. Download it from my web site.
  2. Copy the ospfNeighbors.tcl file to your router's flash (or NVRAM).
  3. Define an alias, for example alias exec ospf tclsh flash:ospfNeighbors.tcl.
Here is a sample printout produced on my lab router:

OSPF neighbors for process ID 1

Router ID Area State Address Interface 0 FULL Serial0/0/0.100 0 FULL/DR FastEthernet0/0

OSPF neighbors for process ID 2

Router ID Area State Address Interface 2 FULL Serial0/1/0
Add comment

Practice the PIX and ASA configuration in a remote lab

If you're studying for your CCSP exam or have to test some of the new features available on PIX and ASA, the remote lab exercises supporting the Securing Networks with PIX and ASA course from Cisco might be just the right thing for you. You'll be able to configure firewall and VPN features of PIX/ASA, as well as test its integration in a network, for example, usage of AAA server and deployment of WebVPN. The lab exercises also cover interesting improvements like transparent firewall, virtual firewall and active/active failover.

More information is available here.

Add comment

Increased number of OSPF processes in MPLS VPN environments

When we were writing the MPLS and VPN architectures books, there was a limit on the number of OSPF processes you could configure per PE-router. The limit was based on the fact that IOS supports up to 32 sources of routing information; two of them are static and connected plus you need an IGP and BGP in the MPLS VPN backbone, resulting in 28 OSPF processes that could be configured on a single PE-router. This "feature" severely limited OSPF-based MPLS VPN deployments until IOS release 12.3(4)T when the limitation was removed, resulting in availability of up to 30 routing processes per VRF.

RIP, BGP and EIGRP never experienced the same limitations as you configure VRF-specific routing instances within address families of a single routing protocol

The details of the actual change that took place are not obvious from the IOS documentation; they simply changed the global allocation of routing process identifiers to per-routing-table identifiers. For example, in the following show ip protocol summary printouts, the two OSPF processes (one in global routing table, one in VRF test) share the same index.

router#show ip protocols summary
Index Process Name
0 connected
1 static
2 ospf 112
router#show ip protocols vrf Test summary
Index Process Name
0 connected
1 static
2 ospf 111
Add comment

Get creative: Is anyone using a default route?

I've got a great question from one of my readers: I have two central sites (primary data center and a disaster recovery data center) and I've inherited a situation where there was a lot of static routing and a default route pointing from the primary campus to the DRC. I've replaced most of the static routing with dynamic routing protocol, but as the documentation is scarce, I am afraid to remove the default static route (which I would need for proper Internet access). Is there a way to figure out whether the default route is still used?

Obviously, there are two types of solutions:
  • The deterministic one: inspect the IP routing tables on primary router and DRC router and identify any IP prefix on the DRC router that has no matching or less specific prefix on the primary router.
  • The non-deterministic ones: try to figure out if any packet is using the default route
    • This is where you can get really creative: how would you figure out if a packet going from the primary data center to the DRC site is using the default route?
see 8 comments

Logging to flash disk

Cisco IOS release 12.4(15)T brought (among a plethora of voice features) the logging to non-volatile storage, a nice-sounding name for the ability to write syslog messages into files on your flash memory (or an embedded disk, if you have one). To configure it, use the logging persistent [url directory] [size filesystem-size] [filesize logging-file-size] global configuration command:
  • The directory argument specifies where you want the files to be stored (for example, flash:/logging).
  • The filesystem-size specifies the maximum disk space the logging files can consume (once you exceed the limit, the oldest file is deleted)
  • The logging-file-size parameter specifies the maximum size of each file (once the file grows too large, a new file is created).

Note: You can store the log files on the router's flash memory if it appears as a disk file system (check with the show file systems command). Wouldn't it be great if this feature would also work on USB drives ...

see 4 comments

DNS resolver package for IOS Tcl

I've ported the dns package of the Tcl standard library to Cisco IOS. You can download it from my web site and install it on your router in just a few steps:
  1. Extract all the files from the ZIP archive and copy the Tcl files into a subdirectory on your router's flash (I would recommend you use flash:tcllib/dns).
  2. Configure the package initialization script with the scripting tcl init flash:tcllib/dns/pkgIndex.tcl global configuration command
To test the successful installation, start the Tcl shell from the command prompt and try to load the DNS package:
router(tcl)#package require dns
see 2 comments

Static routing with Catalyst 3750: and the winner is …

The Static routing with Catalyst 3750 post has generated a lot of good, creative ideas. Some of the proposed solutions were better than the others and some were simply not implementable (but nonetheless, had great creative potential :). Here is my list of the favorites:

A routing protocol: as a few of you have rightly pointed out, this is the best choice.

Aggressive Unidirectional Link Detection (UDLD): this is my second favorite, as it's a reliable link-level mechanism that will detect a break in the fiber cable … exactly the right tool for the job.

Object tracking and reliable static routes would also work. This was my initial solution, but I was worried about its support in Catalyst IOS images. In the meantime, one of the readers has noted that the reliable static routing (or at least the configuration command) works in IOS release 12.2(37)SE, so this might be a viable solution.
GRE tunneling with GRE keepalives probably has performance issues (I am not sure GRE is ASIC-switched on Catalyst 3750). The idea is to create a GRE tunnel between IP addresses on the primary fiber link. If the connectivity breaks, but the subnet remains available, the GRE keepalives will detect the failure.

Spanning tree will not work. It does not test the two-way connectivity and might actually create a loop (if I stop receiving spanning tree hellos, I might assume the link is connected to a workstation and OK to use).

Etherchannel and LACP would most likely fail. If the radio adapters work as switches, so you cannot establish an Etherchannel across the two links. I am also not sure that the PAgP or LACP would detect a unidirectional link when the carrier is present on both ends. Any experience?

Bidirectional Forwarding Detection (BFD) is not available on Catalyst 3750 at all (the ISR routers got it as late as IOS release 12.4(15)T). Furthermore, BFD (as implemented in IOS today) detects a routing protocol neighbor failure, not an IP next-hop failure, so you need to run a routing protocol first (in which case we wouldn't be discussing this scenario, as we would have changed the boss, as someone has suggested).

Edited on September 9th to clearly indicate what would or would not work

see 8 comments

IP Event Dampening

The September IP Corner article, Increase the Stability of your Network, describes the IP Event Dampening, an IOS feature that allows you to disable a flapping interface that continuously disrupts your routing protocol. As always, I went a step beyond that and tested how you can use the Embedded Event Manager to track other network instabilities (for example, a failing OSPF neighbor) and react to them.
Add comment

Workaround: track the actual IP routing status of an interface

In a previous post, I've described how the track interface ip routing command reports incorrect interface state if you use IP Event Dampening feature. To track the actual IP routing readiness of an interface, you could use the following workaround:
  • Create a static IP route pointing to the interface you want to test. Make sure this route is not redistributed into any routing protocols.
  • Track the reachability of the static route
For example, to track the IP routing status of the Serial0/1/0 interface on which the dampening is configured, use the following configuration:
interface Serial0/1/0
dampening 30
ip route Serial0/1/0
track 1 ip route reachability
see 1 comments

Get creative: static routing with Catalyst 3750

I've probably got this question due to my IP Corner article Small site multi-homing, where I've been describing reliable static routing. Here's the scenario: we have two sites, each using a Catalyst 3750 switch, and routing between them using static routes. There's a primary fiber link between them and we're using twisted-pair-to-fiber converters due to port limitations on Cat3750. These converters do not report fiber link down status correctly (the carrier is still present on twisted pair even if fiber is down), so the primary Ethernet interfaces do not go down if the fiber link breaks and the primary static route is not removed, requiring manual action to switch over to the backup link.

My initial reaction was a polite answer explaining that the dynamic routing protocols were invented to handle scenarios like this one, but the poor guy responded that his boss does not want to hear about a dynamic routing protocol.” The next idea was the reliable static routing, tracking next-hop availability over both interfaces, but the Catalyst IOS does not support that, as it's based on 12.2 release.

I've got a few other ideas in the meantime (at least one of them working perfectly), but let's hear it from you first ... what would be your solution to this problem?
see 27 comments

Persistent DHCP bindings stored in NVRAM

If you'd like to implement persistent DHCP bindings on Cisco IOS, but cannot store them on an external server, you could always use the on-board NVRAM. Simply configure ip dhcp database nvram:dhcp.txt. Later on, you can examine the contents of the dhcp.txt file with more nvram:dhcp.txt command.

This post was written in 2007, when a lot of low-end Cisco routers still shipped with flash formatted in the “old” Cisco format and the flash was not really usable to store ever-changing files. For more details on storing DHCP bindings in onboard flash, read the Flash-based DHCP Database blog post.

see 9 comments

Using Tcl packages on Cisco IOS

Although it's not exactly trivial, you can use standard Tcl packages with Tcl
shell on Cisco IOS by following this procedure:

$ tclsh
% pkg_mkIndex . *.tcl
% ^Z
  • Edit the pkgIndex.tcl file created with the pkg_mkIndex command and set the $dir variable to the IOS directory before the first package command (for example, set dir "flash:tcl/").
  • Alternatively, add the Tcl command set dir [file dirname [info script]] in front of the first package command. This command sets the $dir variable to the path of the pkgIndex.tcl file.
  • Transfer all the source files into a directory on the router's flash (or any other local storage device).
  • Configure the execution of the pkgIndex.tcl file at tclsh startup with the scripting tcl init configuration command (for example, scripting tcl init flash:tcl/pkgIndex.tcl).

When you have completed these steps, the pkgIndex.tcl file will be executed every time the Tcl shell is started in Cisco IOS, defining all the packages you've prepared. Now you can use the package require name Tcl command to load the packages you need in your Tcl script.

see 2 comments

Don't miss the obvious

I've recently replaced my old home router (well, actually a combination of two low-end models, one could handle ISDN and the other one 3DES) with a 1812. After I've struggled past the “interesting” interface names (it has 8 switched ports, named FastEthernet2 to FastEthernet9) and brushed up my BVI/VLAN skills, configuring it was a breeze … only the DHCP server was causing me problems; every time my laptop would wake from the standby mode, it would take almost half a minute before it got the LAN IP address. The obvious suspect (as I've installed the 12.4(15)T on it) was the software, the next one DHCP ping timers.

After replacing the software (didn't help) and tweaking DHCP timers (no change), it finally dawned on me: the ethernet ports are switched, so the spanning tree was playing tricks with me. Disabling spanning tree with the spanning-tree portfast interface configuration command solved the problem.
see 13 comments

Correction: do not use the name option of the ip route command

In my February IP corner article, Small Site Multihoming, I've used an obscure name string option of the ip route configuration command to force the router to accept multiple otherwise identical static routes (plus it seemed like a nice way to document what the static route does). While this option is totally harmless on the point-to-point serial links that I was using, one of the readers experienced hard-to-diagnose problems on upstream LAN interfaces that disappeared when we've removed the name option from the configuration.

As the solution presented in the article does not need the name option to differentiate between the static routes (the track keyword is enough to make a difference), it should be removed (and we've already removed it from the HTML and PDF version of the article).

Update: It turned out the problems my reader experienced had nothing to do with the name option of the ip route command, but the generic advice still applies: don't use the features you don't need.

see 5 comments