Your browser failed to load CSS style sheets. Your browser or web proxy might not support elliptic-curve TLS

Building network automation solutions

9 module online course

Start now!

New look

What do geeks do when they wait for Santa? At least some of them tweak their Blogger template ... and so my blog got a completely new look-and-feel (after I've spent three months managing to postpone this task). Any opinions and suggestions are most welcome; after all, I'm not even close to a web designer.
see 2 comments

Best of 2007

Out of curiousity, I've checked which posts generated the most traffic throughout 2007. Here they are:

I've considered only the individual post pages, the blog's home page has 20 times more hits than the next page and most of the top-10 pages are the per-category listings.

And here are the five most commented topics:

Add comment

Please don't ask me “internal” questions

Every now and then I get a question asking about information that is clearly Cisco-internal, for example:

Hello Ivan!

Can you share some information about this IOS:

  • CallGen - bulk call generator
  • Pagent - Packet Generator

I don't know where to get these images and how to get the License Key for Pagent.

Questions like this should always be directed to your Cisco contacts (try the non-support channels, the always overloaded TAC team usually cannot help you). Even if I had the Cisco-internal information you're looking for, I'm bound by so many Non-Disclosure Agreements that I couldn't share it with you.

see 5 comments

OSPF Remote Lab Exercises

Have you ever wanted to practice with all the aspects of the OSPF technology, from simple single-area scenarios to a complex MPLS environment? The new remote lab product we've just released gives you the opportunity to test a number of OSPF concepts and configuration techniques. The Open Shortest Path First - Complete Technology lab bundle is a collection of exercises taken from standard Cisco courses (BSCI, MPLS and IP6FD) enhanced by specific OSPF scenarios (Non-Broadcast Multi-Access, Sham link support) and complete real-life deployment scenario (OSPF Superlab), enabling you to gain advanced skills in configuring and monitoring OSPF in complex and diverse network environments.
Add comment

Display open TCP and UDP ports

With the introduction of Control Plane Policing features (available from 12.3(4)T), you can easily inspect all the open ports (servers and clients) on a router with the show control-plane host open-ports command, resulting in a printout very similar to the netstat -a printout on a Unix/Windows workstation.For example, on the router where I've configured BGP, HTTP server, NTP and DHCP, this command produces the following output (a session to a BGP neighbor as well as a telnet session was established):
R1#show control-plane host open-ports
Active internet connections (servers and established)
Prot Local Address Foreign Address Service State
 tcp *:23 *:0 Telnet LISTEN
 tcp *:80 *:0 HTTP CORE LISTEN
 tcp *:179 *:0 BGP LISTEN
 tcp *:179 BGP ESTABLIS
 tcp *:23 Telnet ESTABLIS
 udp *:67 *:0 DHCPD Receive LISTEN
 udp *:68 *:0 BootP client LISTEN
 udp *:123 *:0 NTP LISTEN
  • This show command does not display non-TCP/UDP servers (OSPF, EIGRP, RSVP) or even some UDP-based services (RIP).
  • Although I was considering writing about CPP for a long time, Artur Szymanski was the one that brought this command to my attention. Thanks!
see 10 comments

What is a BGP RIB failure

Sometimes you'll see a weird route status (RIB-failure) in your BGP table, for example:

GW#show ip bgp ¦ include r>
r> 0 0 65001 i

A more thorough investigation of the BGP entry does not give you a lot of additional information:

GW#show ip bgp
BGP routing table entry for, version 7
Paths: (1 available, best #1, table Default-IP-Routing-Table, RIB-failure(17))
Flag: 0x820
  Advertised to update-groups:
        1 2
  65001 from (
      Origin IGP, metric 0, localpref 100, valid, external, best

The “mistery” is solved when you inspect the entry in the IP routing table:

GW#show ip route
Routing entry for
  Known via "static", distance 1, metric 0 (connected)
  Routing Descriptor Blocks:
  * directly connected, via Null0
      Route metric is 0, traffic share count is 1

The GW router has a static route that collides with the EBGP route and thus the BGP route cannot be inserted in the IP routing table (as the static route has administrative distance 1).

Let's conclude with a few interesting facts about the RIB failures:

  • The RIB failure feature was introduced in IOS release 12.2T; prior to that, the BGP routes with higher administrative distance than other route sources were silently ignored (similar to all other routing protocols).
  • You can display BGP routes that are not inserted in the IP routing table with the show ip bgp rib-failure command, which also explains why the BGP route was not inserted in the IP routing table.
  • The BGP routes that are not used due to higher administrative distance are still advertised to all BGP peers (contrary to what most other distance-vector routing protocols do), unless you configure bgp suppress-inactive (introducted in 12.2T and 12.0(26)S).
see 26 comments

EEM CLI patterns are not context sensitive

When writing EEM applets or policies that act on CLI commands, keep in mind that the pattern matching is not context sensitive. For example, if you want to disable the reload command and use the EEM applet …
event manager applet NoReload
 event cli pattern "reload" sync no skip yes
… you cannot enter the action x.y reload configuration command any more (or any other command that includes the string reload).

To distinguish the reload command from other appearances of the same string, use the ^reload pattern (reload occuring at the beginning of the line).

Trivia: this actually occured to me when I was testing the setup described in the December IP Corner article. Sometimes we have to learn the hard way :)

see 6 comments

Making the case for Layer 2 and Layer 3 VPNs

Occasionally someone would try to persuade me that the layer-2 VPN services are like aspirin (you know, totally harmless plus it could get rid of all your headaches). OK, that might be true if you take the layer-2 VPN offering as a pure transport solution and plug in an extra router (sometimes also called a layer-3 switch by marketing people) between the Service Provider's Ethernet (or whatever they give you) and your LAN. But there are people who don't know the details and plug the SP Ethernet straight into their L2 switch … and things might even work for a while … until the whole network collapses.

In my opinion, we need both L2 and L3 VPN services, but it's important that they are positioned and deployed correctly. You can read more about my views on this topic in the SearchTelecom article Making the case for Layer 2 and Layer 3 VPNs.

The list of all articles I wrote for SearchTelecom is available in the CT3 wiki.

Add comment

Running BGP across parallel serial links

Whenever I'm describing the idea of running BGP across parallel serial links with duplicate IP addresses (like I did in the November IP Corner article, Load Balancing in BGP Networks, section External BGP Load Balancing), there's always someone asking “does it really work?” … so I'm enclosing a tested working configuration.
AS 11AS 12
interface Serial1/1
 ip address
 encapsulation ppp
interface Serial1/2
 ip address
 encapsulation ppp
router bgp 11
 bgp log-neighbor-changes
 neighbor remote-as 12
interface Serial1/1
 ip address
 encapsulation ppp
interface Serial1/2
 ip address
 encapsulation ppp
router bgp 12
 bgp log-neighbor-changes
 neighbor remote-as 11
ip route Null0
Here are a few printouts. First the BGP neighbors …
AS11#show ip bgp summary ¦ begin Neighbor
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 4 12 13 12 2 0 0 00:09:02 1
… then the BGP routing table …
R2#show ip bgp | begin Network
   Network Next Hop Metric LocPrf Weight Path
*> 0 0 12 i
… and finally the internal details of the CEF entry (that's the only way to actually verify that the load balancing is taking place):
AS11#show ip cef internal, version 35, epoch 0, per-destination sharing
0 packets, 0 bytes
  tag information from, shared
    local tag: 17
  via, 0 dependencies, recursive
    next hop, Serial1/1 via
    valid adjacency
    tag rewrite with Se1/1, point2point, tags imposed: {}
  Recursive load sharing using
  Load distribution: 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 (refcount 2)
  Hash OK Interface Address Packets Tags imposed
  1 Y Serial1/1 point2point 0 none
  2 Y Serial1/2 point2point 0 none
  3 Y Serial1/1 point2point 0 none
  4 Y Serial1/2 point2point 0 none
  5 Y Serial1/1 point2point 0 none
  6 Y Serial1/2 point2point 0 none
  7 Y Serial1/1 point2point 0 none
  8 Y Serial1/2 point2point 0 none
  9 Y Serial1/1 point2point 0 none
  10 Y Serial1/2 point2point 0 none
  11 Y Serial1/1 point2point 0 none
  12 Y Serial1/2 point2point 0 none
  13 Y Serial1/1 point2point 0 none
  14 Y Serial1/2 point2point 0 none
  15 Y Serial1/1 point2point 0 none
  16 Y Serial1/2 point2point 0 none
see 11 comments

MPLS Traffic Engineering without a Link State routing protocol

You've probably heard the joke about the honest salesmen: it's not that they're lying, what they know isn't true. I had a similar problem recently: in the 10 MPLS traffic engineering myths and half truths I wrote “Half-truth: MPLS TE only works with OSPF and IS-IS routing protocols.” Ivan Kuchin understood that as “You can run MPLS TE without OSPF or IS-IS.” Although I haven't written that anywhere, I also thought that was the case … so let me try to weasel out of this mess.

I remember being involved in a situation years ago (around the 12.0T release) where someone wanted to use MPLS TE without IS-IS (which was the only supported protocol in those days) and somehow the solution was to set up tunnels using explicit paths, where you have to specify hop-by-hop IP addresses. When you think about it, it makes perfect sense: if you list every IP address in the path, there is no need for constraint-based path calculation (PCALC). However, as it turns out, the later additions to MPLS TE (loose source routing, address exclusion, inter-area MPLS TE, inter-AS MPLS TE) changed the IOS code sufficiently that even the hop-by-hop tunnels cannot be set up without operational OSPF or IS-IS:
  • In order to have MPLS TE running on a router, you need an MPLS TE router-id, and you can only specify that in OSPF or IS-IS routing protocol.
  • Even though the hop-by-hop explicit path is static, the router wants to run PCALC for every hop in the path. If the next-hop IP address is not in the OSPF topology database, the router will not even try to set up the tunnel.

If you want to run MPLS TE in your network, you thus need to run OSPF or IS-IS, even though you might not want to use them for IP packet forwarding. For example, you could enable one of them only on the links actually used for MPLS TE and set the distance to 255 to prevent their routes from getting into the IP routing table (and I've tested it in the lab before writing this post).

see 6 comments

Mandatory EEM CLI commands

The action cli commands used in EEM applets as well as the cli* Tcl functions used in EEM Tcl policies open a virtual Telnet session to a VTY line to execute the CLI commands. The first command you have to execute in the EEM applet is thus the enable command to ensure the next commands will be executed with privilege level 15.

You don't have to specify the enable password.

Likewise, if you want to configure the router, the next command to execute is the configure terminal command, followed by the configuration commands.

The actual execution of the EEM CLI commands can be debugged with the debug event manager action cli command. For example, when the following EEM applet is executed …

event manager applet TEST
 event none
 action 1.0 cli command "enable"
 action 1.1 cli command "configure terminal"
 action 2.0 cli command "interface loopback 3"
 action 2.1 cli command "no shutdown"
… it produces this debugging output:
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : CTL : cli_open called.
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : GW>
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : IN : GW>enable
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : GW#
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : IN : GW#configure terminal
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : Enter configuration commands, one per line.
 End with CNTL/Z.
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : GW(config)#
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : IN : GW(config)#interface loopback 3
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : GW(config-if)#
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : IN : GW(config-if)#no shutdown
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : GW(config-if)#
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : CTL : cli_close called.
see 2 comments

OSPF graceful shutdown

I've described the OSPF Graceful Shutdown functionality in an earlier post and got an excellent question about the implications of using OSPF stub router functionality with external routes.

The more I was digging into this issue, the more design questions I've got … and finally ended up writing a whole IP Corner article about it. You can read the in-depth discussion of design and implementation aspects of OSPF stub router functionality in the December IP Corner article: Bring your Network Closer to Five Nines with Graceful Shutdown.
Add comment

BGP fast session deactivation also speeds up session establishment

You might have been there before: the BGP neighbor becomes reachable after you fix a fault in the network, but the BGP session takes “forever” to be established (actually, the hold off is less than a minute, but time is running slower when you are waiting for the network to recover). However, when testing the BGP fast peering session deactivation, I made an interesting discovery: the restart time is improved as well; as soon as the path to the BGP neighbor appears in the IP routing table, the BGP session is established. The debugging printouts from my router are included below (I've used neighbor fall-over configuration command):
03:28:42: RT: add via, ospf metric [110/75]
03:28:42: RT: NET-RED
03:28:44: RT: Try lookup less specific, default 1
03:28:44: RT: Found subnet on less specific
03:28:44: %BGP-5-ADJCHANGE: neighbor Up
see 8 comments

Recovering from disabled password recovery might not be possible

IOS release 12.3T (and 12.4) introduced a great security feature: the ability to disable password recovery (using the well-known break key sequence) with the no service password-recovery global configuration command. However, once you configure this feature on some routers, you might have no means whatsoever to get it under control if you forget the password.

The IOS documentation states that you should be able to erase NVRAM (thus losing the config, but protecting the password integrity) if you press the break key a few seconds after the Image text-base: 0x........, data-base: 0x........ message appears. Unfortunately, that does not work on the router I've been doing my tests on (2811 with c2800nm-advipservicesk9-mz.124-6.T.bin and ROMMON Version 12.4(1r)). There was simply no way to erase NVRAM, so the router would remain locked up if I had really forgotten the enable password.

Note: After my tests, I was told that pressing the break key as soon as the router is powered up might work.

Moral of the story: test whether you can recover the router with your particular combination of IOS/ROMMON versions before disabling password recovery (and forgetting the password).

see 33 comments

Execute CLI commands with prompts in EEM

In response to my post about combining Tcl shell with EEM to get around the “no prompts” limitation of EEM action cli command, Xavier proposed using the undocumented pattern option of the action cli command, which changes the string the EEM script is expecting to indicate that the current command has been executed.

By default, the EEM action cli command waits until it receives exec-level prompt from the VTY (Router> or Router#), resulting in an endless wait and aborted EEM applet in IOS release 12.4(15)T (earlier releases would hang a VTY line forever) if a CLI command returns an additional prompt. With the pattern option, you can change the expected reply to whatever prompt the CLI command is outputting.For example, to clear interface counters, you could use the following EEM applet:
event manager applet test
 event none
 action 0.9 cli command "enable"
 action 1.0 cli command "clear counter loopback 0" pattern "confirm"
 action 1.1 cli command "y"
When it's executed with the event manager run test command and the event manager CLI debugging is enabled, you'll see the following printout:
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : CTL : cli_open called.
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : OUT : R1>
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : IN : R1>enable
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : OUT : R1#
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : IN : R1#clear counter loopback 0
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : IN : y
%CLEAR-5-COUNTERS: Clear counter on interface Loopback0 by on vty0 (EEM:test)
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : OUT : y
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : OUT : R1#
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : CTL : cli_close called.
see 5 comments

BGP without MPLS?

Designing and operating large BGP networks has always been a challenge, as you have to deploy BGP on all core routers and design a hierarchy of internal BGP routers to get around the full-mesh limitation. When MPLS was introduced, it gave us means of deploying BGP only on the network edges, with the core routers carrying just the information about the BGP next hops.

As I know some of you run large networks, could you help me understand what you're using (without giving away too much information, of course):
  • Are you running a BGP network without MPLS or are you using BGP on the edges and MPLS transport in the core?
  • If you have a large number of BGP routers, do you have a nice hierarchy of BGP route reflectors (or confederations) or ad-hoc implementation where every router has all neighbors as RR-clients?

Full disclosure: I might use the information you give me in an upcoming article.

see 5 comments

Feed format conclusions

You've generated a lot of comments when I've asked you about your opinion on the feed format I should use. Thank you for your time and efforts!

Although the majority doesn't care what feed format I use, there are a few that would prefer the full feed … and I can understand that completely, as I am also reading a lot of content in my RSS reader. Therefore, I will not change the feed format until I will find means of giving you enough text in the shortened feed to enable you to make a decision as to whether the original post is worth reading.

Last but not least, I'll have to reduce the post frequency in the next few weeks due to other work I have to complete in this year and I'll vanish (temporarily) around Christmas, to be back in the first half of January.
Add comment

SNRS labs on available on Partner E-learning Connection

If you want to study for your CCSP certification and have partner-level access to Cisco's web, you can schedule Securing Networks with Cisco Routers and Switches remote labs free-of-charge straight from Partner E-learning connection by clicking this link (partner-level CCO username required).

If you're not a Cisco partner, you can buy the same labs from our web site.
Add comment

BGP fast session deactivation

We all know that BGP is meant to converge slowly ... well, the MPLS/VPN service providers tend to disagree, as their users are not used to minute-long convergence times. One of the major components of slow BGP convergence is the time it takes a router to discover that a neighbor has disappeared. Traditionally, the BGP keepalive packets were sent every minute and it took up to three minutes to discover that a neighbor is down. Of course you could fine-tune those times with the neighbor timers configuration command, but the reduced timers resulted in increased TCP traffic and consequently increased CPU load, which could reach tens of percents if the timers were set to a few seconds and the router had lots of BGP neighbors.

The neighbor loss detection has improved dramatically in 12.3T and 12.0S with the introduction of the fast session deactivation, where a BGP session is dropped as soon as the route to the BGP neighbor is lost. You can configure this feature with the ominous-sounding neighbor fall-over configuration command. Obviously, this feature does not work well if you use default routing (or summaries), since the path to the BGP neighbor is never completely lost. In that case, you can use a route-map option of the neighbor fall-over command (introduced in 12.4(4)T) to select which less specific route is still a valid route to the BGP neighbor.Here are the logging and debugging printouts from a router that lost a BGP neighbor and discovered it after the BGP hold time has expired:
00:00:48: %BGP-5-ADJCHANGE: neighbor Up
00:01:23: RT: del via, ospf metric [110/129]
00:01:23: RT: delete subnet route to
00:03:49: %BGP-3-NOTIFICATION: received from neighbor 4/0 (hold time expired) 0 bytes
00:03:49: %BGP-5-ADJCHANGE: neighbor Down BGP Notification received
As you can see, there is more than a two minute gap between the time the OSPF route to the BGP neighbor was lost and the time the BGP session went down (and the BGP routes were recalculated). When the neighbor fall-over is configured, the BGP session is disconnected as soon as the OSPF route to the neighbor is gone:
00:08:12: RT: del via, ospf metric [110/75]
00:08:12: RT: delete subnet route to
00:08:12: RT: NET-RED
00:08:12: RT: Try lookup less specific, default 1
00:08:12: RT: Failed found subnet on less specific
00:08:12: RT: return NULL
00:08:12: %BGP-5-ADJCHANGE: neighbor Down Route to peer lost

I will cover the route-map option as well as the design implications of this feature in an upcoming IP corner article.

see 2 comments

Configure DNS servers through IPCP

After I've fixed the default routing in my home office, I've stumbled across another problem: the two ISPs I'm using for my primary and backup link have DNS servers that reply solely to the DNS requests sent from their own IP address range:

When the traffic is switched from the primary to the backup ISP, I therefore also need to switch the DNS servers. Fortunately, this is quite easy to do on a router; you just need to configure ppp ipcp dns request on the dialer interface and the router starts asking for the DNS server address as part of the IPCP negotiation.
The negotiation process can be debugged with the debug ppp negotiation command; it's a bit more complex than usual in my case since the access server has no secondary DNS (only the primary DNS is configured):
Se1/0 IPCP: O CONFREQ [Closed] id 1 len 22
Se1/0 IPCP: Address (0x030600000000)
Se1/0 IPCP: PrimaryDNS (0x810600000000)
Se1/0 IPCP: SecondaryDNS (0x830600000000)

Se1/0 PPP: Process pending ncp packets
Se1/0 IPCP: Redirect packet to Se1/0
Se1/0 IPCP: I CONFREQ [REQsent] id 1 len 10
Se1/0 IPCP: Address (0x03060A000021)
Se1/0 IPCP: O CONFACK [REQsent] id 1 len 10
Se1/0 IPCP: Address (0x03060A000021)
Se1/0 IPCP: I CONFREJ [ACKsent] id 1 len 10
Se1/0 IPCP: SecondaryDNS (0x830600000000)
Se1/0 IPCP: O CONFREQ [ACKsent] id 2 len 16
Se1/0 IPCP: Address (0x030600000000)
Se1/0 IPCP: PrimaryDNS (0x810600000000)
Se1/0 IPCP: I CONFNAK [ACKsent] id 2 len 16
Se1/0 IPCP: Address (0x03060A000022)
Se1/0 IPCP: PrimaryDNS (0x81060A00000A)
Se1/0 IPCP: O CONFREQ [ACKsent] id 3 len 16
Se1/0 IPCP: Address (0x03060A000022)
Se1/0 IPCP: PrimaryDNS (0x81060A00000A)
Se1/0 IPCP: I CONFACK [ACKsent] id 3 len 16
Se1/0 IPCP: Address (0x03060A000022)
Se1/0 IPCP: PrimaryDNS (0x81060A00000A)

Se1/0 IPCP: State is Open
The results can be inspected only with the show host command:
GW#show host
Default domain is not set
Name/address lookup uses domain service
Name servers are
The access server receiving the call requires no special configuration; the first IP address configured with the ip name-server command is used as the primary DNS and the second one as the secondary. Alternatively, you can configure a different set of DNS servers to pass to the client with the ppp ipcp dns primary-DNS-address secondary-DNS-address interface configuration command.

Unfortunately, the integration with LAN clients is not as seamless as with DHCP; to make the whole solution work, you have to configure the router as a forwarding DNS server and make the LAN clients use the router as the default gateway and DNS server with the DHCP pool configuration:
ip dns server
ip dhcp pool LAN
   import all
see 8 comments

Enhanced show interfaces command

It's amazing how many options (most of them still undocumented) the show interfaces command accepts in IOS release 12.4T (I won't even start guessing when each one was introduced, if you're running old IOS releases, please feel free to comment):

  • show interfaces description displays interface names, L1 and L2 status (line and line-protocol status) and interface description. Extremely handy if you want to check which interfaces are up/down.
  • show interfaces counters protocol status displays the L3 protocols active on each interface.
  • show interfaces summary displays the state of various interface queues and related drop counters in a nice tabular format.
  • show interfaces accounting displays per-protocol in/out counters.

Here are a few sample printouts:

read more see 9 comments

Can I combine EEM applets with Tcl shell?

When I've been describing the limitations of kron, ??? quickly asked an interesting question: “as I cannot insert extra input keystrokes with EEM applet, can I run a Tcl script from it with the action sequence cli command "tclsh script" command and use the typeahead function call to get around the limitation?” The only answer I could give at that time was “maybe” … and obviously it was time for a more thorough test. The short result is: YES, you can do it (at least in IOS release 12.4(15)T1).

… and here is the long description of the test. I've started by creating a small Tcl script (see below) that clears the counters on Loopback 0. As the clear counters command requires keyboard input and generates a syslog message, it was a perfect test case.

typeahead "y"
exec "clear counter loop 0"

I've copied this script into the flash:tcl/clearL0.tcl and tested it:

R1#tclsh flash:tcl/clearL0.tcl
%CLEAR-5-COUNTERS: Clear counter on interface Loopback0 by console

So far, so good. Next, I've created an EEM applet with no trigger …

event manager applet Clear
 event none
 action 1.0 cli command "enable"
 action 1.1 cli command "tclsh flash:tcl/clearL0.tcl"

… enabled the EEM CLI debugging and started it:

R1#debug event man action cli
Debug EEM action cli debugging is on
R1#event man run Clear
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : CTL : cli_open called.
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : OUT : R1>
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : IN : R1>enable
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : OUT : R1#
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : IN : R1#tclsh flash:tcl/clearL0.tcl
%CLEAR-5-COUNTERS: Clear counter on interface Loopback0 by on vty0 (EEM:Clear)
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : OUT : R1#
%HA_EM-6-LOG: Clear : DEBUG(cli_lib) : : CTL : cli_close called.

Great. It works. Let's move on: I've inserted a trigger into the EEM applet that ran the applet when an OSPF neighbor reached FULL adjacency:

event manager applet Clear
 event syslog pattern "OSPF-5-ADJCHG.*to FULL"
 action 1.0 cli command "enable"
 action 1.1 cli command "tclsh flash:tcl/clearL0.tcl"

And now for the final test: after I've enabled the serial interface, OSPF neighbors established adjacency …

R1(config-if)#interface ser 1/0
R1(config-if)#no shutdown
%LINK-3-UPDOWN: Interface Serial1/0, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/0, changed state to up
%OSPF-5-ADJCHG: Process 1, Nbr on Serial1/0 from LOADING to FULL, Loading Done
%CLEAR-5-COUNTERS: Clear counter on interface Loopback0 by on vty0 (EEM:Clear)

… and the counters on Loopback0 were cleared. Test completed :)

You can find more Tclsh-related information in the Tclsh on Cisco IOS tutorial. Sample Tclsh scripts are available in the Tclsh script library. If you need expert help in planning, developing or deploying Tclsh scripts in your network, contact the author.

This article is part of You've asked for it series.

see 16 comments

Hands-on MPLS Traffic Engineering

I've written a lot about MPLS Traffic Engineering (not nearly as much as I would like, but there are always time constraints), as I believe this technology has interesting applications in Enterprise networks (and we all know that a lot of Service Providers are using it anyway). You might have seen my 10 MPLS Traffic Engineering Myths or the Perfect Load Balancing article … and if you don't know what I'm talking about, there's always the introductory Traffic Engineering the Service Provider Network.

The major problem of MPLS TE is that it's complex and that networking engineers usually lack the hands-on skills, and this is where we can help you: we've just rolled out the revised MPLS TE lab exercises. Compared to remote lab offerings from other sources, these lab exercises are very focused: you get step-by-step instructions (but no recipes, that would spoil the learning process), preconfigured equipment (so you don't have to configure IP addresses or IP routing protocols to get the job done) and detailed solutions explaining which task is achieved using a specific set of configuration commands.

I was able to get a discount for my readers: if you click this link and type in the promotion code 42B078 (expires on January 15th, 2008), you'll get a one week subscription to the MPLS TE remote lab bundle for €56. As this is a subscription offering, you can run the lab exercises as often as you like within a week of the purchasing date. And if you need one more argument to be persuaded, check the lab topology; you can experiment in a preconfigured nine router network :)
Add comment

Your opinion matters: full versus short blog feed

Major headacheUnfortunately, I've started discovering the downsides of blogging: someone is actively stealing my content and publishing it on a number of blogs. I don't mind people quoting me (even larger passages of my text) as long as the proper credits appear somewhere in the post, but straight copy from my blog feed is too much.

Obviously, the easy stopgap measure would be to change the feed format from full to short, which would only pass the first few lines of every blog post into the feed, resulting in a bit more effort on your end, as you'd have to click on the post link in the feed reader to view the whole text. Would this be a major nuisance for anyone?
see 20 comments

Kron: poor-man's cron

When two groups within Cisco needed time-based command execution in Cisco IOS, they (in a typical big-corporation fashion) decided to implement the same wheel from two different sets of spokes and rims. One group built the Embedded Event Manager with its event timer cron command (introduced in 12.2(25)S and 12.3(14)T), the other group created the more limited kron command set (introduced in 12.3(1)).

EEM is almost a perfect superset of kron, both can trigger a set of CLI commands at reload, at periodic intervals or at certain time in the future. The only extra functionality offered by kron is the ability to specify a different username for each event (whereas all EEM applets have to run under a common username) … and kron is available in older IOS releases.

Similar to EEM applets, CLI commands executed within kron cannot expect extra input (so you cannot execute clear counters or reload from kron) and the output they generate is lost unless you use output filters to redirect it to an external file.

Here is a simple configuration that archives the router's running configuration every sunday half an hour before midnight:

kron policy-list archiveConfig
 cli archive config
kron occurrence archiveConfig at 23:30 Sun recurring
 policy-list archiveConfig
see 5 comments

Enable password or enable secret?

I've stumbled across a blog post that indicates there's still confusion on some fundamental configuration issues. I will not even try to guess whether there is a wide consensus on how to configure a router, but these are the facts (and here is a ten year old position from Cisco):
see 11 comments

Emulate dialup links with serial lines

I had to figure out various PPP parameters (and associated Cisco IOS behavior) and didn't have real dial-up equipment in my lab setup. I could have gone with PPPoE, but it turned out it's way simpler to emulate dialup connections (at least the PPP negotiations work as expected) on fixed serial lines. This is the minimum setup you need on the “caller” side …
interface Serial1/0
 ip address negotiated
 encapsulation ppp
 ppp authentication pap optional
 ppp pap sent-username client password 0 client
… and this is the “server”-side configuration:
interface Serial1/0
 ip address
 encapsulation ppp
 peer default ip address
 ppp authentication pap callin
username client password client
To trigger PPP negotiations, shut down and re-enable the serial interface on either side.

Note: As I'm using PAP authentication, I could use the more secure username secret configuration command, which would not work with CHAP.

see 8 comments

Simple SMTP server

When I was testing the SMTP client on the routers (configured with action … mail command or as part of EEM SMTP library), I didn't want the messages with weird addresses and content to circulate through our e-mail system (plus I wanted to see the results immediately), so I wrote a simple SMTP server in Perl that prints the messages it receives. It accepts a single parameter: the IP address to listen on (only needed on workstations with multiple IP addresses).

You need the Net::SMTP::Server module on top of the standard Perl distribution to make it work.

You can download the PERL source code or compiled EXE file (for Windows users) from CT3 wiki.

see 8 comments

IPv6 deployment: Time for action?

A while ago I was asked to write an article about IPv6 training. I could just cover the training aspect, like what's offered (answer: not much) and whether someone can train the whole operations team like you could in the IPv4 or MPLS/VPN world (answer: no), but I wanted to understand whether anyone is really using IPv6 in a production network. I found a few academic networks (after all, there are about 2000 IPv6 prefixes assigned and someone should be doing something with them), but not much of what I would call a real production environment, which is a bad thing, as it looks like the IPv4 address space will get saturated in a few years.

Update 2010-03-12: Numerous commercial ISPs now offer native IPv6 connectivity, but they also face significant deployment challenges. You will find an overview of those in my Market trends in Service Provider networks workshop (register for the online webinar). Advanced backbone designs and configurations are explained in the Building IPv6 Service Provider core workshop (register for the online webinar).

read more see 8 comments

BGP default route

Update 2011-12-07: You might also want to read the Responsible generation of BGP default route article describing the ISP side of the solution.
Update 2008-08-10: IOS behavior has changed; fixed the article.

Martin Kluge sent me an interesting BGP question: he has two upstream links and runs BGP on both. Since his router is low on RAM, he cannot accept full routing, so he's just announcing his IP prefix and using static default routing toward upstream ISPs.

The relevant configuration on the GW router is somewhat similar to the configuration I've used as a staring point in my lab:

read more see 24 comments

Type 7 decryption in Cisco IOS

Tim Riegert sent me an interesting hint: you don't need password crackers to decode type-7 passwords, you just need access to a router. Here's how you do it:

We'll turn on type-7 encryption for local passwords and generate a test username

R1(config)#service password-encryption
R1(config)#username test password t35t:pa55w0rd

Next we'll inspect the generated username with the show running command

R1(config)#do show run | include username
username test password 7 08351F1B1D431516475E1B54382F

Now we'll create a key chain and enter the type-7 encrypted password as the key string …

R1(config)#key chain decrypt
R1(config-keychain)#key 1
R1(config-keychain-key)#key-string 7 08351F1B1D431516475E1B54382F

… and the show command does the decryption for us.

R1(config-keychain-key)#do show key chain decrypt
Key-chain decrypt:
    key 1 -- text "t35t:pa55w0rd"
        accept lifetime (always valid) - (always valid) [valid now]
        send lifetime (always valid) - (always valid) [valid now]
see 12 comments

Show active IOS processes

You can use the show process cpu sorted command in combination with an output filter to display only those IOS processes that consumed noticeable amount of CPU time in the last five minutes, last minute or last five seconds. Use the following patterns to construct your regular expression:
  • The [0-9.]+% pattern will match any non-zero percentage;
  • The 0.00% pattern will obviously match the zero-percentage display;
  • As the percentage figures are separated by various amounts of whitespace characters, we have to use the ' +' pattern to match those;
The show filter should exclude the processes that have the zero percentage in the desired column and any percentage in the other two columns (any other filter would show too many or too few processes). To display processes active in the last minute, use the show process cpu sorted 1min | exclude [0-9.]+% +0.00% +[0-9.]+% command (and define an alias to make it easier to use).You could use these configuration commands to define the aliases:
alias exec cpu1min show process cpu sorted 1min | exclude [0-9.]+% +0.00%
alias exec cpu5sec show process cpu sorted 5sec | exclude 0.00% +[0-9.]+% +[0-9.]+%
alias exec cpu5min show process cpu sorted 5min | exclude [0-9.]+% +[0-9.]+% +0.00%
A sample printout from one of my routers is included:
CPU utilization for five seconds: 4%/0%; one minute: 2%; five minutes: 2%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
5 27260472 1470452 18538 0.00% 1.74% 1.78% 0 Check heaps
62 536 226 2371 3.27% 0.52% 0.15% 2 Virtual Exec
30 248000 230369 1076 0.16% 0.07% 0.02% 0 IP Input
25 617780 25736 24004 0.00% 0.03% 0.00% 0 Per-minute
43 32 485 65 0.00% 0.01% 0.00% 0 TCP Timer
Add comment

Persistent EEM variables

Someone has asked me a while ago whether it's possible to retain variable values between invocations of an EEM policy. Since a new copy of Tcl interpreter is started for each event, global variables obviously won't work; they are lost as soon as the Tcl policy is finished. A potential solution is to modify the router's configuration and save the values you wish to preserve in event manager environment, but that's a time-consuming process that interferes with whatever router configuration management process you have.

The real solution is based on the appl_setinfo and appl_reqinfo calls. They work, but like many other Tcl-related IOS features they are … well … weird.This time, the programmers managed to implement WORO (Write-Once-Read-Once) memory:
  • The value you want to preserve is saved with appl_setinfo key name data value function call. Keys must be unique; you can only set the same key once. If you try to set the value of a key multiple times, the function does not overwrite the previous value but fails.
  • You can read the value with appl_reqinfo key name function call. If the key value hasn't been set, it returns an empty string and sets the $_cerrno variable, otherwise it returns a list with 'data' as the first element and your value as the second list element (I have to admit I've seen simpler APIs :).
  • Once you read the key value, it's gone. You cannot read it twice.
If all this sounds a bit strange, don't worry; here's a working example:
::cisco::eem::event_register_cli sync no skip no pattern "show"

namespace import ::cisco::eem::*
namespace import ::cisco::lib::*

Set the variable value to zero (in case we haven't saved the value before) and read the previous value

set lastCnt 0
set getLastCnt [ appl_reqinfo key "showCounter" ]

If the first element in the list is 'data', then the second element is our value.

if { [ lindex $getLastCnt 0 ] == "data" } {
  set lastCnt [ lindex $getLastCnt 1 ]

Increase the counter and generate a syslog message

incr lastCnt
action_syslog priority info msg "Show command was executed $lastCnt times"

Save the new value of the counter to be retrieved by the next invocation of the same policy.

appl_setinfo key "showCounter" data $lastCnt

This article is part of You've asked for it series.

Add comment

Ones are slower than zeroes

Thinking about the implications of bit stuffing I wrote about in the SDLC post, I realized that long sequences of ones would be transmitted slower than long sequences of zeroes due to an extra bit being inserted after every fifth consecutive one. The theory would predict a 20% decrease in transmission speed.

Of course I wanted to test this phenomenon immediately. I needed real equipment with low-speed serial links (that would make the difference more pronounced and less dependent on other intra-router delays), so I started one of the BGP lab exercises; Basic BGP Setup looked like a perfect choice. We're using 64 kbps Frame Relay links in the lab with a Frame Relay switch in the middle (makes the task of designing an arbitrary WAN topology quite simple), so it's a perfect environment to test this thing. And, not surprisingly, the results confirmed the theory:
Internal-Core#ping data 0000 size 1200 repeat 50
Sending 50, 1200-byte ICMP Echos to, timeout is 2 seconds:
Packet has data pattern 0x0000
Success rate is 100 percent (50/50), round-trip min/avg/max = 608/608/632 ms
Internal-Core#ping data FFFF size 1200 repeat 50
Sending 50, 1200-byte ICMP Echos to, timeout is 2 seconds:
Packet has data pattern 0xFFFF
Success rate is 100 percent (50/50), round-trip min/avg/max = 724/724/728 ms

The results are almost too close to the predicted ones, but they are real :)

see 3 comments

Impact of Netflow accounting

A few days ago I was describing the impact of CEF accounting on a router (based on its architecture). The picture is clearer with Netflow: Cisco has published a white paper detailing the impact of various types of Netflow accounting on a large variety of platforms, from an 1800 ISR to the GSR (12000).

The link to this white paper has been published in Joe Harris' blog.

see 5 comments

Load balancing with BGP

A while ago, people believed you cannot do load balancing with BGP (they also believed the Earth was flat a few years before that). While that's no longer true, designing good BGP load balancing is still a complex undertaking. In the November IP Corner article, Load Balancing in BGP Networks I'm describing almost all options you have to implement BGP-based load balancing, both within your autonomous system as well as across an AS boundary.
see 8 comments

Catch Skype with Flexible Packet Matching

Joe Harris published an excellent post detailing how you can use Flexible Packet Matching to recognize (and potentially block) Skype traffic. The solution depends on recognizing the first four bytes sent by the Skype application in a TCP session. While this is a great idea, you have to be aware that there's always a non-zero chance of false positives, more so as the described filter is testing the beginning of the payload in every TCP packet (not just the first data packet in the session).
see 3 comments

Impact of CEF accounting

Jozef Janitor wrote a highly relevant comment to my post on CEF accounting: enabling it on a Catalyst switch drastically reduces its performance. The impact of CEF accounting (or other forwarding plane features) depends on switching implementation:
  • There is almost no impact on single-CPU software platforms; the router has to perform CEF lookup anyway and increases the CEF accounting counters on-the-fly;
  • Distributed software platforms are more complex, as the central CPU has to (at the very least) collect the switching statistics.
  • The impact on hardware platforms is dependent on the layer 3 lookup implementation
In the case of a Catalyst 3550 switch, the L3 TCAM it uses to support IP routing, PBR and access-lists is so different from the CEF data structures that IOS has to fall back to software L3 switching to collect CEF accounting data.
Add comment

Back to the roots: it all started with SDLC

My recent post about problems with old modems has generated a lot of comments with some very useful ideas, but nobody addressed the question “why was a long string of ones not a problem?”, so let's start there. Almost all WAN synchronous protocols in use today are descendants of venerable SDLC invented by IBM more than 30 years ago. SDLC was later extended to support connectionless and balanced modes, resulting in HDLC. PPP is just an extension of HDLC, adding support for negotiations and standard layer-3 protocol demultiplexing. In SDLC, IBM also solved the frame delimiting and associated escape character problem inherent in previous protocols like BSC (DLE was used in BSC) by introducing bit stuffing: a zero would be inserted after five consecutive ones (and silently removed by the receiver) to differentiate the regular data stream from framing (six consecutive ones) and abort (more than six consecutive ones) sequences. Thus, the HDLC (or PPP) data stream can never contain more than six consecutive ones and the long sequences of ones never cause synchronization loss.

IBM obviously also had problems with bad modems and solved it with the NRZI encoding that was part of SDLC standard (and a major pain in the good old days when the appliques on the old Cisco routers did not support it and we've been trying hard to penetrate IBM accounts). You can still configure NRZI encoding on most routers' serial links (it might depend on the actual hardware platform) with the nrzi-encoding interface configuration command (you had to do it with jumpers in the AGS+). Incidentally, changing interface encoding to NRZI was really helpful when you had to break things in the preparation for the troubleshooting part of the original CCIE lab).

Enough theory, let's summarize the proposed solutions:
  • The nrzi-encoding (if available) is the best one, as it reliably solves the problem, is transparent and does not incur additional overhead.
  • Compression or encryption are OK, but they result in significant CPU overhead (unless you have hardware encryption/compression modules) and might (at least in theory) still produce a long sequence of zeroes, although with a very low probability. IPSec also introduces overhead due to additional IPSec headers.
  • LFI (effectively multilink PPP over a single link) is also a good solution, as the PPP framing and MLPPP headers break the long sequences of zeroes (you might have to fine-tune the fragment size with ppp multilink fragment size configuration command), but it introduces overhead on the WAN link.
  • IP fragmentation would work, but would be quite bandwidth-consuming. If the fragmentation would be performed by the router, the overhead would be 20 bytes per fragment (IP header), if the sending host performs the fragmentation, the overhead is 40 bytes per fragment for TCP sessions. For example, if we reduce the IP MTU size to 256 bytes, the TCP session overhead is over 18% (and we were scoffing at the ATM designers that made us live with 10% overhead).
There were also a few suggestions that would not work very well:
  • The invert data command would only help if the modem has problems with long strings of zeroes, not with long strings of the same value.
  • The tunnel key command just sets a 4-byte field in the GRE header but does not affect the encapsulated data at all.
see 3 comments

After a year

I've started publishing posts on a regular basis about a year ago … and this week you've raced by another milestone: more than 1000 page loads/day (on average; weekdays are more active and weekends are a bit slow). For those of you who like statistics, here's the weekly report …

IOS Hints Weekly Statistics
… and the last few months:

IOS Hints Monthly Statistics
see 5 comments

React to excessive jitter with EEM

William Chu sent me a working configuration he uses to measure jitter with the IP SLA tool and react to excessive jitter on the primary link. First you have to create the jitter probe with the IP SLA commands:

ip sla monitor 3000
 type jitter →
   dest-ipaddr dest-port 12333 →
   source-ipaddr codec g729a →
   codec-numpackets 100
 tos 184
 frequency 10

Note: The continuation character (→) indicates that the configuration command spans multiple lines

Next you have to define the IP SLA reaction to excessive jitter. William configured his router to react when the jitter exceeds 300 milliseconds and returns back to normal when the jitter falls below 290 milliseconds (some hysteresis is always a good thing).

ip sla monitor reaction-configuration 3000 →
  react MOS threshold-value 300 290 →
  threshold-type consecutive →
  action-type trapOnly

As the last step in the SLA configuration, you have to start the probe:

ip sla monitor schedule 3000 →
  life forever start-time now

After the SLA probe and out-of-bounds reaction have been configured, the router will generate syslog messages whenever the jitter gets above the threshold as well as when it falls below the second threshold. You can then use the EEM applets to act on the syslog messages:

event manager applet MOS-Below
 event syslog occurs 1 period 120 →
   pattern "Threshold below for MOS"
 ... actions ...
event manager applet MOS-Above
 event syslog occurs 1 period 120 →
   pattern "Threshold exceeded for MOS"
 ... actions ...
see 4 comments

Learning Tcl

If you're new to Tcl and would like to start using it on Cisco IOS, here's what worked for me:

  • I've downloaded the ActiveTcl from ActiveState. It's always good to have development environment on your workstation.
  • The documentation that comes with ActiveTcl is quite cryptic, but once you know what you're looking for, it's OK.
  • To get to the "I know what I'm looking for" stage, I've used Tcl/Tk: A Developer's Guide.

If you know better beginner books/sources, let us all know.

You can find more Tclsh-related information in the Tclsh on Cisco IOS tutorial. Sample Tclsh scripts are available in the Tclsh script library. If you need expert help in planning, developing or deploying Tclsh scripts in your network, contact the author.

This article is part of You've asked for it series.

see 2 comments

For the oldtimers: swamped with zeroes

In the pre-DSL days, you had two options to get a short-haul high-speed link (at least in Europe): take E1 (or fractional E1) from a telecom (which was more expensive than a highway robbery, as the cost was recurring) or use baseband modems with proprietary encoding techniques on physical copper wires (assuming you could get them). As it turned out, some of these encoding techniques were not as good as the others (but the equipment was relatively cheap, so the budget limits usually forced the decision). We had our own share of modem-related problems, but they were never as bad as what I've heard from one of my students: his modems would lose synchronization when transmitting a long string of zeroes over a regular synchronous serial interface; ping ip size 1000 data 0000 would be enough to bring down the link.

And now two questions for you:
  • What could you do on the router to fix this problem?
  • Why was the synchronization retained when transmitting a long string of ones?
see 11 comments

MPLS Traffic Engineering myths

Did you believe MPLS TE was a quality-of-service feature? Did someone persuade you it's mandatory to run OSPF or IS-IS if you want to deploy MPLS TE? I've collected a few more myths like these two and explained the actual facts behind them in an article published by SearchTelecom.

The list of all articles I wrote for SearchTelecom is available in the CT3 wiki.

Add comment

Download router configurations via TFTP

In a previous post, I've described how you can turn your router into a TFTP server. As you can configure the router to serve any file residing on it, you can also pull startup and running configuration from it with TFTP, providing that you configure:
tftp-server nvram:startup-config
tftp-server system:running-config

Warning: Due to total lack of any security features in TFTP protocol, use this functionality only in lab environment.

see 5 comments

Remove the configuration prompt

I should probably write this one on April 1st, but maybe October 31st is not such a bad choice after all … if you configure no service prompt config, the configuration prompt is gone; when you enter the configuration mode with the configure terminal command, you get an empty line (like you did with Cisco software release 9.1 some 15 years ago). Similarly, you can disable command-line editing with the no editing line configuration command or terminal no editing exec-level command. If only there would be a way to disable the context-sensitive help :)
see 9 comments

More details on OSPF route filters

I did a few follow-up tests with the distribute-list in OSPF configuration command and stumbled across a few interesting facts (IOS release 12.4(15)T1 on a 3725 platform):
  • Although the router allows you to configure distribute-list acl in interface, it does not work. Routes received through that interface (or having the interface as the next-hop) are not filtered.
  • When you apply the distribute-list in command, the routing table is not changed. Clearing the IP routing table does not help, you have to clear ALL OSPF processes (including bringing down all OSPF adjacencies) with the clear ip ospf process command for the route filter to take effect.
  • The same limitations don't apply in the other direction: when you remove the distribute-list in, SPF is triggered and the routes appear in the IP routing table automatically.
  • The somewhat undocumented gateway option of the distribute-list in command works, but not quite as I would expect: the IP next hop, not the router-ID of the router advertising the IP prefix is matched by the prefix-list.

And, last but not least, I've lab-verified my previous claim: applying the distribute-list in on a transit router can result in a black hole, as the LSAs themselves are not filtered.

see 9 comments

Send an e-mail when an interface goes down

John S. Pumphrey recently asked an interesting question: “Can the router send an e-mail when an interface goes down?” The enterprisey solution is obvious: deploy a high-end EMS to collect SNMP traps and use its API to write a custom module that would use a MQ interface to alert the operator. Fortunately, Event Manager applets in Cisco IOS provide action mail command (available in 12.3(14)T and 12.4) that can send an e-mail to a SMTP server straight from the router.

There are two ways you can detect that an interface went down with EEM: either you track the interface status with a track object and start an EEM applet when the track object changes state or you catch the syslog messages reporting that the interface line protocol changed state to down. The second approach is obviously more generic, as a single applet can act on multiple interfaces.

event manager applet MailOnIfDown
 event syslog occurs 1 →
    pattern "LINEPROTO-5-UPDOWN.*to down" →
    period 1


  • If you want to limit the applet to serial interfaces only, you could change the pattern to LINEPROTO-5-UPDOWN.*Serial.*to down.
  • The → continuation character is used to indicate that a single configuration line has been split to increase readability.

The action mail command specifies the mail server's address (use a hostname and DNS lookup or ip host configuration command to make the EEM applet more generic), from and to address, message subject and body. In each of these fields, you can use EEM environment variables that you can define with the event manager environment configuration command. Each EEM event also defines a few environment variables that you can use (see the table of EEM system-defined variables on CCO). For example, you can define the e-mail recipient in the router's configuration and use the _syslog_msg variable to include the syslog message in the e-mail body:

event manager environment _ifDown_rcpt [email protected]
event manager applet MailOnIfDown
 event syslog occurs 1 →
    pattern "LINEPROTO-5-UPDOWN.*to down" →
    period 1
 action 1.0 mail server "mail-gw" →
    to "$_ifDown_rcpt" from "[email protected]" →
    subject "Interface down on R1" →
    body "$_syslog_msg"

You can make the applet even more generic with the help of action info type routername command, which stores the current router's name into the $_info_routername environment variable:

event manager environment _ifDown_rcpt [email protected]
event manager applet MailOnIfDown
 event syslog occurs 1 →
    pattern "LINEPROTO-5-UPDOWN.*to down" →
    period 1
 action 1.0 info type routername
 action 2.0 mail server "mail-gw" →
    to "$_ifDown_rcpt" from "$" →
    subject "Interface down on $_info_routername" →
    body "$_syslog_msg"

Note: This article is part of You've asked for it series.

see 3 comments

Display a summary of OSPF interfaces

When I was lab-testing the ideas for my article on EIGRP-to-OSPF migration, I had to check the OSPF interface cost ... and complained immensely at the amount of information produced by the show ip ospf interface command. Luckily I've double-checked before starting to write a Tcl script and realized that a brief keyword was added in IOS release 12.3, producing exactly the printout I was looking for.On my test router, the show ip ospf interfaces brief command produced this nicely-tabulated printout:
a1#show ip ospf interface brief
Interface PID Area IP Address/Mask Cost State Nbrs F/C
Lo0 1 0 1 LOOP 0/0
Se0/0/0.100 1 0 50 P2P 1/1
Fa0/0 1 0 1 BDR 1/1
Fa0/1 1 11 1 DR 0/0

The netmask-format of your line has to be set to bit-count for this command to work properly, otherwise the columns will be shifted to the right.

see 1 comments

Debugging cached CEF adjacencies

A while ago I wrote about cached CEF adjacencies and the impact they have on ARP caching. If you ever need to, you can debug them with the debug ip cef table command. As this command might produce a lot of output in a production network, always use it in combination with an access-list that limits the debugging to the selected address range.

Alternatively, you can use the debug arp adjacency command, but you cannot limit its output with an access-list

For example, to test cached CEF adjacencies in subnet, I've used the following commands:
rtr#show ip access-list 99
Standard IP access list 99
10 permit, wildcard bits (26 matches)
rtr#debug ip cef table 99
IP CEF table debugging is on for access list 99
rtr#debug arp
ARP packet debugging is on

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to, timeout is 2 seconds:

08:57:27: IP ARP: creating incomplete entry for IP address: interface FastE
08:57:27: IP ARP: sent req src 0016.c876.8b38,
dst 0000.0000.0000 FastEthernet0/0
08:57:27: IP ARP: rcvd rep src 000c.29a7.8ade, dst FastEthernet0/0
08:57:27: CEF-IP: Checking dependencies of
08:57:27: CEF-Table: Adjacency-prefix add request -- succee
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/4 ms
Add comment

OSPF neighbors stuck in EXSTART

This problem is quite rare, but tantalizing enough to warrant mentioning: OSPF neighbors are forever stuck in EXSTART state (occasionally going to DOWN and back to EXSTART).

I've actually stumbled across it accidentally in my lab and have luckily seen it before, so I knew immediately what it was.

The moment you start suspecting that something might be wrong with the OSPF adjacencies and use debug ip ospf adj command, the problem becomes obvious: the Database Description packet contains an Interface MTU field and if the value received from the neighbor is higher than the IP MTU configured on the inbound interface, the DBD packet is rejected (section 10.6 of the RFC 2328). The router with the lower MTU complains that “Nbr x.x.x.x has larger interface MTU” and the other router moans about protocol violations (First DBD and we are not SLAVE).

As always, there are two ways to solve this problem:

  • The correct one: fix the MTU issues;
  • The other one: disable MTU checks with the ip ospf mtu-ignore interface configuration command (which might be OK as long as the hardware is able to receive oversized packets and the router is not using fixed-size input buffers).
see 16 comments

GRE tunnel keepalives

The IP-over-IP (usually GRE) tunnels (commonly in combination with IPSec to provide security) are frequently used when you want to transport private IP traffic over public IP network that does not support layer 3 VPNs. If you use the GRE tunnels in combination with default routing (or route summarization), you can get serious routing issues when the tunnel destination disappears, but a default (or summary) route in the IP routing table still covers it. You could work around this issue by deploying a routing protocol over the GRE tunnel (which could lead to hard to diagnose routing loops if you're not careful) or by using GRE keepalives introduced in IOS release 12.2(8)T.

The implementation of the GRE keepalives is amazing: the router sending the keepalive packet constructs a GRE packet that would be sent from the remote end back to itself (effectively building a GRE reply), sets the GRE protocol type to zero (to indicate the keepalive packet) and sends the whole packet through the tunnel (effectively encapsulating GRE reply into another GRE envelope). The receiving router strips the GRE envelope and routes the inside packet … which is the properly formatted GRE keepalive reply.

This trick allows you to implement different GRE keepalive timers on each end of the link. For example, the remote site might use fast keepalive timers to detect loss of primary link and switch over to a backup link, while the central site would use less frequent keepalive tests to detect failed remote site (if there is a single path to the remote site, you don't care too much when you detect it's down).

Every ingenious solution has its drawbacks and this one is no exception: if the receiving router protects its IP addresses (to stop spoofing attacks), it will drop the incoming GRE keepalive packet. Furthermore, a document available on Cisco's web describes the issues of using GRE keepalives in IPSec environment.
see 12 comments

Tabular display of interface MTUs

When I started exploring the details of MTU handling in Cisco IOS, I quickly got tired of analyzing various long printouts to extract the MTU sizes, so I wrote a Tcl script that display hardware, IP and MPLS MTUs in a tabular format. To install it on your router:
  1. Download it from my web site and copy it to your router's flash or NVRAM.
  2. Define an alias, for example alias exec mtu tclsh flash:displayMTU.tcl.

The script recognizes two parameters: the ip parameter displays only the interfaces that have IP configured and the mpls parameter displays only the MPLS-enabled interfaces.

Here is a sample printout from one of my lab routers:
Interface Hardware IP MPLS
FastEthernet0/0 1500 1600
FastEthernet0/1 1500
Serial1/0 1500 1500
Serial1/1 1380 1380 1430
Serial1/2 1500 1500
Serial1/3 1500
Loopback0 1514 1514
Tunnel0 1514 1476 1476

R2#mtu mpls
Interface Hardware IP MPLS
FastEthernet0/0 1500 1600
Serial1/1 1380 1380 1430
Tunnel0 1514 1476 1476
You can find more Tclsh-related information in the Tclsh on Cisco IOS tutorial. Sample Tclsh scripts are available in the Tclsh script library. If you need expert help in planning, developing or deploying Tclsh scripts in your network, contact the author.
see 2 comments

ICND1 blended learning

If you're a regular reader of my blog, you probably won't need this one, but it might be useful to your colleagues who are reaching for the first rung on the Cisco certification ladder.

If your first thought was CCNA, you're wrong; Cisco came up with a sub-CCNA certification called Cisco Certified Entry Networking Technician.

We've just rolled out a blended e-learning version of the requisite ICND1 (yes, that is ICND part one) course containing student guides in e-learning format and all the associated remote lab exercises (performed on actual devices, no simulations).
Add comment

Get creative: OSPF route filters

A few of the readers pointed out that I've forgotten about OSPF route filters in the post I wrote about running OSPF across the firewall. Actually I didn't, as it's only available in IOS, not in ASA/PIX. However, this is not the familiar route update filtering available in EIGRP, BGP or RIP. The route updates (LSAs) are not filtered (as they should not be in a link-state protocol); the filters control the transfer of the IP prefixes from the SPF tree to the IP routing table.

To keep the record straight, I have to point out that this is not a new feature. You could have achieved similar results (almost) forever with the distance 255 command (remember: 255 = totally unbeliavable = not installed). Later on, you could use the distribute-list in router configuraton command and in 12.2T this command has been extended to accept a route-map.

However, as this functionality is totally different from the distance vector route filters and doesn't play well with OSPF concepts, you have to use it very carefully. Every OSPF router still propagates all the LSAs, but if it fails to install the resulting IP prefixes in the IP routing table, it's a potential blackhole (see the following figure).

With all this being said, I cannot see the “OSPF inbound filtering” as it's slightly inappropriately named being widely used. Do you have any great scenarios where this feature would be really helpful?
see 7 comments

The tale of the three MTUs

An IOS device configured for IP+MPLS routing uses three different Maximum Transmission Unit (MTU) values:

  • The hardware MTU configured with the mtu interface configuration command
  • The IP MTU configured with the ip mtu interface configuration command
  • The MPLS MTU configured with the mpls mtu interface configuration command

The hardware MTU specifies the maximum packet length the interface can support … or at least that's the theory behind it. In reality, longer packets can be sent (assuming the hardware interface chipset doesn't complain); therefore you can configure MPLS MTU to be larger than the interface MTU and still have a working network. Oversized packets might not be received correctly if the interface uses fixed-length buffers; platforms with scatter/gather architecture (also called particle buffers) usually survive incoming oversized packets.

IP MTU is used to determine whether a non-labeled IP packet forwarded through an interface has to be fragmented (the IP MTU has no impact on labeled IP packets). It has to be lower or equal to hardware MTU (and this limitation is enforced). If it equals the HW MTU, its value does not appear in the running configuration and it tracks the changes in HW MTU. For example, if you configure ip mtu 1300 on a Serial interface, it will appear in the running configuration as long as the hardware MTU is not equal to 1300 (and will not change as the HW MTU changes). However, as soon as the mtu 1300 is configured, the ip mtu 1300 command disappears from the configuration and the IP MTU yet again tracks the HW MTU.

The MPLS MTU determines the maximum size of a labeled IP packet (MPLS shim header + IP payload size). If the overall length of the labeled packet (including the shim header) is greater than the MPLS MTU, the packet is fragmented. The MPLS MTU can be greater than the HW MTU assuming the hardware architecture and interface chipset support that (and the router will warn you that you might be getting into trouble). Similar to the ip mtu command, the mpls mtu command will only appear in the running configuration if the MPLS MTU is different from the HW MTU. However, contrary to the behavior of the IP MTU, any change in HW MTU with the mtu configuration command also resets the MPLS MTU to HW MTU.

The behavior as described above was tested on a 3725 router running IOS release 12.4(15)T1. Although the MPLS MTU Command Changes document claims that you cannot set MPLS MTU larger than then interface MTU from IOS release 12.4(11)T, I was still able to do it in 12.4(15)T1.

see 13 comments