Building network automation solutions

9 module online course

Start now!

Blog Posts in December 2007

Best of 2007

Out of curiousity, I've checked which posts generated the most traffic throughout 2007. Here they are:

I've considered only the individual post pages, the blog's home page has 20 times more hits than the next page and most of the top-10 pages are the per-category listings.

And here are the five most commented topics:

add comment

Please don't ask me “internal” questions

Every now and then I get a question asking about information that is clearly Cisco-internal, for example:

Hello Ivan!

Can you share some information about this IOS:

  • CallGen - bulk call generator
  • Pagent - Packet Generator

I don't know where to get these images and how to get the License Key for Pagent.

Questions like this should always be directed to your Cisco contacts (try the non-support channels, the always overloaded TAC team usually cannot help you). Even if I had the Cisco-internal information you're looking for, I'm bound by so many Non-Disclosure Agreements that I couldn't share it with you.

see 5 comments

OSPF Remote Lab Exercises

Have you ever wanted to practice with all the aspects of the OSPF technology, from simple single-area scenarios to a complex MPLS environment? The new remote lab product we've just released gives you the opportunity to test a number of OSPF concepts and configuration techniques. The Open Shortest Path First - Complete Technology lab bundle is a collection of exercises taken from standard Cisco courses (BSCI, MPLS and IP6FD) enhanced by specific OSPF scenarios (Non-Broadcast Multi-Access, Sham link support) and complete real-life deployment scenario (OSPF Superlab), enabling you to gain advanced skills in configuring and monitoring OSPF in complex and diverse network environments.
add comment

What is a BGP RIB failure

Sometimes you'll see a weird route status (RIB-failure) in your BGP table, for example:

GW#show ip bgp ¦ include r>
r> 0 0 65001 i

A more thorough investigation of the BGP entry does not give you a lot of additional information:

GW#show ip bgp
BGP routing table entry for, version 7
Paths: (1 available, best #1, table Default-IP-Routing-Table, RIB-failure(17))
Flag: 0x820
  Advertised to update-groups:
        1 2
  65001 from (
      Origin IGP, metric 0, localpref 100, valid, external, best

The “mistery” is solved when you inspect the entry in the IP routing table:

GW#show ip route
Routing entry for
  Known via "static", distance 1, metric 0 (connected)
  Routing Descriptor Blocks:
  * directly connected, via Null0
      Route metric is 0, traffic share count is 1

The GW router has a static route that collides with the EBGP route and thus the BGP route cannot be inserted in the IP routing table (as the static route has administrative distance 1).

Let's conclude with a few interesting facts about the RIB failures:

  • The RIB failure feature was introduced in IOS release 12.2T; prior to that, the BGP routes with higher administrative distance than other route sources were silently ignored (similar to all other routing protocols).
  • You can display BGP routes that are not inserted in the IP routing table with the show ip bgp rib-failure command, which also explains why the BGP route was not inserted in the IP routing table.
  • The BGP routes that are not used due to higher administrative distance are still advertised to all BGP peers (contrary to what most other distance-vector routing protocols do), unless you configure bgp suppress-inactive (introducted in 12.2T and 12.0(26)S).
see 26 comments

EEM CLI patterns are not context sensitive

When writing EEM applets or policies that act on CLI commands, keep in mind that the pattern matching is not context sensitive. For example, if you want to disable the reload command and use the EEM applet …
event manager applet NoReload
 event cli pattern "reload" sync no skip yes
… you cannot enter the action x.y reload configuration command any more (or any other command that includes the string reload).

To distinguish the reload command from other appearances of the same string, use the ^reload pattern (reload occuring at the beginning of the line).

Trivia: this actually occured to me when I was testing the setup described in the December IP Corner article. Sometimes we have to learn the hard way :)

see 6 comments

Making the case for Layer 2 and Layer 3 VPNs

Occasionally someone would try to persuade me that the layer-2 VPN services are like aspirin (you know, totally harmless plus it could get rid of all your headaches). OK, that might be true if you take the layer-2 VPN offering as a pure transport solution and plug in an extra router (sometimes also called a layer-3 switch by marketing people) between the Service Provider's Ethernet (or whatever they give you) and your LAN. But there are people who don't know the details and plug the SP Ethernet straight into their L2 switch … and things might even work for a while … until the whole network collapses.

In my opinion, we need both L2 and L3 VPN services, but it's important that they are positioned and deployed correctly. You can read more about my views on this topic in the SearchTelecom article Making the case for Layer 2 and Layer 3 VPNs.

The list of all articles I wrote for SearchTelecom is available in the CT3 wiki.

add comment

MPLS Traffic Engineering without a Link State routing protocol

You've probably heard the joke about the honest salesmen: it's not that they're lying, what they know isn't true. I had a similar problem recently: in the 10 MPLS traffic engineering myths and half truths I wrote “Half-truth: MPLS TE only works with OSPF and IS-IS routing protocols.” Ivan Kuchin understood that as “You can run MPLS TE without OSPF or IS-IS.” Although I haven't written that anywhere, I also thought that was the case … so let me try to weasel out of this mess.

I remember being involved in a situation years ago (around the 12.0T release) where someone wanted to use MPLS TE without IS-IS (which was the only supported protocol in those days) and somehow the solution was to set up tunnels using explicit paths, where you have to specify hop-by-hop IP addresses. When you think about it, it makes perfect sense: if you list every IP address in the path, there is no need for constraint-based path calculation (PCALC). However, as it turns out, the later additions to MPLS TE (loose source routing, address exclusion, inter-area MPLS TE, inter-AS MPLS TE) changed the IOS code sufficiently that even the hop-by-hop tunnels cannot be set up without operational OSPF or IS-IS:
  • In order to have MPLS TE running on a router, you need an MPLS TE router-id, and you can only specify that in OSPF or IS-IS routing protocol.
  • Even though the hop-by-hop explicit path is static, the router wants to run PCALC for every hop in the path. If the next-hop IP address is not in the OSPF topology database, the router will not even try to set up the tunnel.

If you want to run MPLS TE in your network, you thus need to run OSPF or IS-IS, even though you might not want to use them for IP packet forwarding. For example, you could enable one of them only on the links actually used for MPLS TE and set the distance to 255 to prevent their routes from getting into the IP routing table (and I've tested it in the lab before writing this post).

see 6 comments

Mandatory EEM CLI commands

The action cli commands used in EEM applets as well as the cli* Tcl functions used in EEM Tcl policies open a virtual Telnet session to a VTY line to execute the CLI commands. The first command you have to execute in the EEM applet is thus the enable command to ensure the next commands will be executed with privilege level 15.

You don't have to specify the enable password.

Likewise, if you want to configure the router, the next command to execute is the configure terminal command, followed by the configuration commands.

read more see 2 comments

OSPF graceful shutdown

I've described the OSPF Graceful Shutdown functionality in an earlier post and got an excellent question about the implications of using OSPF stub router functionality with external routes.

The more I was digging into this issue, the more design questions I've got … and finally ended up writing a whole IP Corner article about it. You can read the in-depth discussion of design and implementation aspects of OSPF stub router functionality in the December IP Corner article: Bring your Network Closer to Five Nines with Graceful Shutdown.
add comment

BGP fast session deactivation also speeds up session establishment

You might have been there before: the BGP neighbor becomes reachable after you fix a fault in the network, but the BGP session takes “forever” to be established (actually, the hold off is less than a minute, but time is running slower when you are waiting for the network to recover). However, when testing the BGP fast peering session deactivation, I made an interesting discovery: the restart time is improved as well; as soon as the path to the BGP neighbor appears in the IP routing table, the BGP session is established. The debugging printouts from my router are included below (I've used neighbor fall-over configuration command):
03:28:42: RT: add via, ospf metric [110/75]
03:28:42: RT: NET-RED
03:28:44: RT: Try lookup less specific, default 1
03:28:44: RT: Found subnet on less specific
03:28:44: %BGP-5-ADJCHANGE: neighbor Up
see 8 comments

Recovering from disabled password recovery might not be possible

IOS release 12.3T (and 12.4) introduced a great security feature: the ability to disable password recovery (using the well-known break key sequence) with the no service password-recovery global configuration command. However, once you configure this feature on some routers, you might have no means whatsoever to get it under control if you forget the password.

The IOS documentation states that you should be able to erase NVRAM (thus losing the config, but protecting the password integrity) if you press the break key a few seconds after the Image text-base: 0x........, data-base: 0x........ message appears. Unfortunately, that does not work on the router I've been doing my tests on (2811 with c2800nm-advipservicesk9-mz.124-6.T.bin and ROMMON Version 12.4(1r)). There was simply no way to erase NVRAM, so the router would remain locked up if I had really forgotten the enable password.

Note: After my tests, I was told that pressing the break key as soon as the router is powered up might work.

Moral of the story: test whether you can recover the router with your particular combination of IOS/ROMMON versions before disabling password recovery (and forgetting the password).

see 33 comments

Execute CLI commands with prompts in EEM

In response to my post about combining Tcl shell with EEM to get around the “no prompts” limitation of EEM action cli command, Xavier proposed using the undocumented pattern option of the action cli command, which changes the string the EEM script is expecting to indicate that the current command has been executed.

By default, the EEM action cli command waits until it receives exec-level prompt from the VTY (Router> or Router#), resulting in an endless wait and aborted EEM applet in IOS release 12.4(15)T (earlier releases would hang a VTY line forever) if a CLI command returns an additional prompt. With the pattern option, you can change the expected reply to whatever prompt the CLI command is outputting.
read more see 5 comments

BGP without MPLS?

Designing and operating large BGP networks has always been a challenge, as you have to deploy BGP on all core routers and design a hierarchy of internal BGP routers to get around the full-mesh limitation. When MPLS was introduced, it gave us means of deploying BGP only on the network edges, with the core routers carrying just the information about the BGP next hops.

As I know some of you run large networks, could you help me understand what you're using (without giving away too much information, of course):
  • Are you running a BGP network without MPLS or are you using BGP on the edges and MPLS transport in the core?
  • If you have a large number of BGP routers, do you have a nice hierarchy of BGP route reflectors (or confederations) or ad-hoc implementation where every router has all neighbors as RR-clients?

Full disclosure: I might use the information you give me in an upcoming article.

see 5 comments

Feed format conclusions

You've generated a lot of comments when I've asked you about your opinion on the feed format I should use. Thank you for your time and efforts!

Although the majority doesn't care what feed format I use, there are a few that would prefer the full feed … and I can understand that completely, as I am also reading a lot of content in my RSS reader. Therefore, I will not change the feed format until I will find means of giving you enough text in the shortened feed to enable you to make a decision as to whether the original post is worth reading.

Last but not least, I'll have to reduce the post frequency in the next few weeks due to other work I have to complete in this year and I'll vanish (temporarily) around Christmas, to be back in the first half of January.
add comment