Build the Next-Generation Data Center
6 week online course starting in spring 2017

New look

What do geeks do when they wait for Santa? At least some of them tweak their Blogger template ... and so my blog got a completely new look-and-feel (after I've spent three months managing to postpone this task). Any opinions and suggestions are most welcome; after all, I'm not even close to a web designer.

Best of 2007

Out of curiousity, I've checked which posts generated the most traffic throughout 2007. Here they are:

I've considered only the individual post pages, the blog's home page has 20 times more hits than the next page and most of the top-10 pages are the per-category listings.

And here are the five most commented topics:

Please don't ask me “internal” questions

Every now and then I get a question asking about information that is clearly Cisco-internal, for example:

Hello Ivan!

Can you share some information about this IOS:

  • CallGen - bulk call generator
  • Pagent - Packet Generator

I don't know where to get these images and how to get the License Key for Pagent.

Questions like this should always be directed to your Cisco contacts (try the non-support channels, the always overloaded TAC team usually cannot help you). Even if I had the Cisco-internal information you're looking for, I'm bound by so many Non-Disclosure Agreements that I couldn't share it with you.

OSPF Remote Lab Exercises

Have you ever wanted to practice with all the aspects of the OSPF technology, from simple single-area scenarios to a complex MPLS environment? The new remote lab product we've just released gives you the opportunity to test a number of OSPF concepts and configuration techniques. The Open Shortest Path First - Complete Technology lab bundle is a collection of exercises taken from standard Cisco courses (BSCI, MPLS and IP6FD) enhanced by specific OSPF scenarios (Non-Broadcast Multi-Access, Sham link support) and complete real-life deployment scenario (OSPF Superlab), enabling you to gain advanced skills in configuring and monitoring OSPF in complex and diverse network environments.

Display open TCP and UDP ports

With the introduction of Control Plane Policing features (available from 12.3(4)T), you can easily inspect all the open ports (servers and clients) on a router with the show control-plane host open-ports command, resulting in a printout very similar to the netstat -a printout on a Unix/Windows workstation.For example, on the router where I've configured BGP, HTTP server, NTP and DHCP, this command produces the following output (a session to a BGP neighbor as well as a telnet session was established):

R1#show control-plane host open-ports
Active internet connections (servers and established)
Prot Local Address Foreign Address Service State
 tcp *:23 *:0 Telnet LISTEN
 tcp *:80 *:0 HTTP CORE LISTEN
 tcp *:179 *:0 BGP LISTEN
 tcp *:179 BGP ESTABLIS
 tcp *:23 Telnet ESTABLIS
 udp *:67 *:0 DHCPD Receive LISTEN
 udp *:68 *:0 BootP client LISTEN
 udp *:123 *:0 NTP LISTEN
  • This show command does not display non-TCP/UDP servers (OSPF, EIGRP, RSVP) or even some UDP-based services (RIP).
  • Although I was considering writing about CPP for a long time, Artur Szymanski was the one that brought this command to my attention. Thanks!

What is a BGP RIB failure

Sometimes you'll see a weird route status (RIB-failure) in your BGP table, for example:

GW#show ip bgp ¦ include r>
r> 0 0 65001 i

A more thorough investigation of the BGP entry does not give you a lot of additional information:

GW#show ip bgp
BGP routing table entry for, version 7
Paths: (1 available, best #1, table Default-IP-Routing-Table, RIB-failure(17))
Flag: 0x820
  Advertised to update-groups:
        1 2
  65001 from (
      Origin IGP, metric 0, localpref 100, valid, external, best

The “mistery” is solved when you inspect the entry in the IP routing table:

GW#show ip route
Routing entry for
  Known via "static", distance 1, metric 0 (connected)
  Routing Descriptor Blocks:
  * directly connected, via Null0
      Route metric is 0, traffic share count is 1

The GW router has a static route that collides with the EBGP route and thus the BGP route cannot be inserted in the IP routing table (as the static route has administrative distance 1).

Let's conclude with a few interesting facts about the RIB failures:

  • The RIB failure feature was introduced in IOS release 12.2T; prior to that, the BGP routes with higher administrative distance than other route sources were silently ignored (similar to all other routing protocols).
  • You can display BGP routes that are not inserted in the IP routing table with the show ip bgp rib-failure command, which also explains why the BGP route was not inserted in the IP routing table.
  • The BGP routes that are not used due to higher administrative distance are still advertised to all BGP peers (contrary to what most other distance-vector routing protocols do), unless you configure bgp suppress-inactive (introducted in 12.2T and 12.0(26)S).

EEM CLI patterns are not context sensitive

When writing EEM applets or policies that act on CLI commands, keep in mind that the pattern matching is not context sensitive. For example, if you want to disable the reload command and use the EEM applet …

event manager applet NoReload
 event cli pattern "reload" sync no skip yes
… you cannot enter the action x.y reload configuration command any more (or any other command that includes the string reload).

To distinguish the reload command from other appearances of the same string, use the ^reload pattern (reload occuring at the beginning of the line).

Trivia: this actually occured to me when I was testing the setup described in the December IP Corner article. Sometimes we have to learn the hard way :)

Making the case for Layer 2 and Layer 3 VPNs

Occasionally someone would try to persuade me that the layer-2 VPN services are like aspirin (you know, totally harmless plus it could get rid of all your headaches). OK, that might be true if you take the layer-2 VPN offering as a pure transport solution and plug in an extra router (sometimes also called a layer-3 switch by marketing people) between the Service Provider's Ethernet (or whatever they give you) and your LAN. But there are people who don't know the details and plug the SP Ethernet straight into their L2 switch … and things might even work for a while … until the whole network collapses.

In my opinion, we need both L2 and L3 VPN services, but it's important that they are positioned and deployed correctly. You can read more about my views on this topic in the SearchTelecom article Making the case for Layer 2 and Layer 3 VPNs.

The list of all articles I wrote for SearchTelecom is available in the CT3 wiki.

Running BGP across parallel serial links

Whenever I'm describing the idea of running BGP across parallel serial links with duplicate IP addresses (like I did in the November IP Corner article, Load Balancing in BGP Networks, section External BGP Load Balancing), there's always someone asking “does it really work?” … so I'm enclosing a tested working configuration.

AS 11AS 12
interface Serial1/1
 ip address
 encapsulation ppp
interface Serial1/2
 ip address
 encapsulation ppp
router bgp 11
 bgp log-neighbor-changes
 neighbor remote-as 12
interface Serial1/1
 ip address
 encapsulation ppp
interface Serial1/2
 ip address
 encapsulation ppp
router bgp 12
 bgp log-neighbor-changes
 neighbor remote-as 11
ip route Null0
Here are a few printouts. First the BGP neighbors …
AS11#show ip bgp summary ¦ begin Neighbor
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 4 12 13 12 2 0 0 00:09:02 1
… then the BGP routing table …
R2#show ip bgp | begin Network
   Network Next Hop Metric LocPrf Weight Path
*> 0 0 12 i
… and finally the internal details of the CEF entry (that's the only way to actually verify that the load balancing is taking place):
AS11#show ip cef internal, version 35, epoch 0, per-destination sharing
0 packets, 0 bytes
  tag information from, shared
    local tag: 17
  via, 0 dependencies, recursive
    next hop, Serial1/1 via
    valid adjacency
    tag rewrite with Se1/1, point2point, tags imposed: {}
  Recursive load sharing using
  Load distribution: 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 (refcount 2)
  Hash OK Interface Address Packets Tags imposed
  1 Y Serial1/1 point2point 0 none
  2 Y Serial1/2 point2point 0 none
  3 Y Serial1/1 point2point 0 none
  4 Y Serial1/2 point2point 0 none
  5 Y Serial1/1 point2point 0 none
  6 Y Serial1/2 point2point 0 none
  7 Y Serial1/1 point2point 0 none
  8 Y Serial1/2 point2point 0 none
  9 Y Serial1/1 point2point 0 none
  10 Y Serial1/2 point2point 0 none
  11 Y Serial1/1 point2point 0 none
  12 Y Serial1/2 point2point 0 none
  13 Y Serial1/1 point2point 0 none
  14 Y Serial1/2 point2point 0 none
  15 Y Serial1/1 point2point 0 none
  16 Y Serial1/2 point2point 0 none

MPLS Traffic Engineering without a Link State routing protocol

You've probably heard the joke about the honest salesmen: it's not that they're lying, what they know isn't true. I had a similar problem recently: in the 10 MPLS traffic engineering myths and half truths I wrote “Half-truth: MPLS TE only works with OSPF and IS-IS routing protocols.” Ivan Kuchin understood that as “You can run MPLS TE without OSPF or IS-IS.” Although I haven't written that anywhere, I also thought that was the case … so let me try to weasel out of this mess.

I remember being involved in a situation years ago (around the 12.0T release) where someone wanted to use MPLS TE without IS-IS (which was the only supported protocol in those days) and somehow the solution was to set up tunnels using explicit paths, where you have to specify hop-by-hop IP addresses. When you think about it, it makes perfect sense: if you list every IP address in the path, there is no need for constraint-based path calculation (PCALC). However, as it turns out, the later additions to MPLS TE (loose source routing, address exclusion, inter-area MPLS TE, inter-AS MPLS TE) changed the IOS code sufficiently that even the hop-by-hop tunnels cannot be set up without operational OSPF or IS-IS:

  • In order to have MPLS TE running on a router, you need an MPLS TE router-id, and you can only specify that in OSPF or IS-IS routing protocol.
  • Even though the hop-by-hop explicit path is static, the router wants to run PCALC for every hop in the path. If the next-hop IP address is not in the OSPF topology database, the router will not even try to set up the tunnel.

If you want to run MPLS TE in your network, you thus need to run OSPF or IS-IS, even though you might not want to use them for IP packet forwarding. For example, you could enable one of them only on the links actually used for MPLS TE and set the distance to 255 to prevent their routes from getting into the IP routing table (and I've tested it in the lab before writing this post).

Mandatory EEM CLI commands

The action cli commands used in EEM applets as well as the cli* Tcl functions used in EEM Tcl policies open a virtual Telnet session to a VTY line to execute the CLI commands. The first command you have to execute in the EEM applet is thus the enable command to ensure the next commands will be executed with privilege level 15.

You don't have to specify the enable password.

Likewise, if you want to configure the router, the next command to execute is the configure terminal command, followed by the configuration commands.

The actual execution of the EEM CLI commands can be debugged with the debug event manager action cli command. For example, when the following EEM applet is executed …

event manager applet TEST
 event none
 action 1.0 cli command "enable"
 action 1.1 cli command "configure terminal"
 action 2.0 cli command "interface loopback 3"
 action 2.1 cli command "no shutdown"
… it produces this debugging output:
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : CTL : cli_open called.
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : GW>
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : IN : GW>enable
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : GW#
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : IN : GW#configure terminal
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : Enter configuration commands, one per line.
 End with CNTL/Z.
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : GW(config)#
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : IN : GW(config)#interface loopback 3
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : GW(config-if)#
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : IN : GW(config-if)#no shutdown
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : OUT : GW(config-if)#
%HA_EM-6-LOG: TEST : DEBUG(cli_lib) : : CTL : cli_close called.

OSPF graceful shutdown

I've described the OSPF Graceful Shutdown functionality in an earlier post and got an excellent question about the implications of using OSPF stub router functionality with external routes.

The more I was digging into this issue, the more design questions I've got … and finally ended up writing a whole IP Corner article about it. You can read the in-depth discussion of design and implementation aspects of OSPF stub router functionality in the December IP Corner article: Bring your Network Closer to Five Nines with Graceful Shutdown.

BGP fast session deactivation also speeds up session establishment

You might have been there before: the BGP neighbor becomes reachable after you fix a fault in the network, but the BGP session takes “forever” to be established (actually, the hold off is less than a minute, but time is running slower when you are waiting for the network to recover). However, when testing the BGP fast peering session deactivation, I made an interesting discovery: the restart time is improved as well; as soon as the path to the BGP neighbor appears in the IP routing table, the BGP session is established. The debugging printouts from my router are included below (I've used neighbor fall-over configuration command):

03:28:42: RT: add via, ospf metric [110/75]
03:28:42: RT: NET-RED
03:28:44: RT: Try lookup less specific, default 1
03:28:44: RT: Found subnet on less specific
03:28:44: %BGP-5-ADJCHANGE: neighbor Up

Recovering from disabled password recovery might not be possible

IOS release 12.3T (and 12.4) introduced a great security feature: the ability to disable password recovery (using the well-known break key sequence) with the no service password-recovery global configuration command. However, once you configure this feature on some routers, you might have no means whatsoever to get it under control if you forget the password.

The IOS documentation states that you should be able to erase NVRAM (thus losing the config, but protecting the password integrity) if you press the break key a few seconds after the Image text-base: 0x........, data-base: 0x........ message appears. Unfortunately, that does not work on the router I've been doing my tests on (2811 with c2800nm-advipservicesk9-mz.124-6.T.bin and ROMMON Version 12.4(1r)). There was simply no way to erase NVRAM, so the router would remain locked up if I had really forgotten the enable password.

Note: After my tests, I was told that pressing the break key as soon as the router is powered up might work.

Moral of the story: test whether you can recover the router with your particular combination of IOS/ROMMON versions before disabling password recovery (and forgetting the password).

Execute CLI commands with prompts in EEM

In response to my post about combining Tcl shell with EEM to get around the “no prompts” limitation of EEM action cli command, Xavier proposed using the undocumented pattern option of the action cli command, which changes the string the EEM script is expecting to indicate that the current command has been executed.

By default, the EEM action cli command waits until it receives exec-level prompt from the VTY (Router> or Router#), resulting in an endless wait and aborted EEM applet in IOS release 12.4(15)T (earlier releases would hang a VTY line forever) if a CLI command returns an additional prompt. With the pattern option, you can change the expected reply to whatever prompt the CLI command is outputting.For example, to clear interface counters, you could use the following EEM applet:

event manager applet test
 event none
 action 0.9 cli command "enable"
 action 1.0 cli command "clear counter loopback 0" pattern "confirm"
 action 1.1 cli command "y"
When it's executed with the event manager run test command and the event manager CLI debugging is enabled, you'll see the following printout:
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : CTL : cli_open called.
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : OUT : R1>
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : IN : R1>enable
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : OUT :
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : OUT : R1#
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : IN : R1#clear counter loopback 0
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : IN : y
%CLEAR-5-COUNTERS: Clear counter on interface Loopback0 by on vty0 (EEM:test)
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : OUT : y
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : OUT : R1#
%HA_EM-6-LOG: test : DEBUG(cli_lib) : : CTL : cli_close called.

BGP without MPLS?

Designing and operating large BGP networks has always been a challenge, as you have to deploy BGP on all core routers and design a hierarchy of internal BGP routers to get around the full-mesh limitation. When MPLS was introduced, it gave us means of deploying BGP only on the network edges, with the core routers carrying just the information about the BGP next hops.

As I know some of you run large networks, could you help me understand what you're using (without giving away too much information, of course):

  • Are you running a BGP network without MPLS or are you using BGP on the edges and MPLS transport in the core?
  • If you have a large number of BGP routers, do you have a nice hierarchy of BGP route reflectors (or confederations) or ad-hoc implementation where every router has all neighbors as RR-clients?

Full disclosure: I might use the information you give me in an upcoming article.

Feed format conclusions

You've generated a lot of comments when I've asked you about your opinion on the feed format I should use. Thank you for your time and efforts!

Although the majority doesn't care what feed format I use, there are a few that would prefer the full feed … and I can understand that completely, as I am also reading a lot of content in my RSS reader. Therefore, I will not change the feed format until I will find means of giving you enough text in the shortened feed to enable you to make a decision as to whether the original post is worth reading.

Last but not least, I'll have to reduce the post frequency in the next few weeks due to other work I have to complete in this year and I'll vanish (temporarily) around Christmas, to be back in the first half of January.