Building Network Automation Solutions
6 week online course starting in September 2017

ARP table with logical and physical interfaces

In a layer-3 switching environment, the ARP table displayed with the show arp command lists the logical (L3) interfaces, for example the VLAN or BVI interface. This Tcl script displays the logical as well as physical interface associated with each IP/MAC address.

Martin Hecko gave me the idea for this script and helped to test it on a Catalyst switch. Thank you!

You can find more Tclsh-related information in the Tclsh on Cisco IOS tutorial. Sample Tclsh scripts are available in the Tclsh script library. If you need expert help in planning, developing or deploying Tclsh scripts in your network, contact the author.

More NAT caveats

A month ago I wrote about NAT caveats in Cisco IOS release 12.4 that occur when the outside addresses match IP access list or route map used in ip nat inside command. I recently discovered more caveats: if you have an inbound access-list on the outside interface, the packets dropped by the access-list still generate NAT entries (and might result in a denial-of-service attack when the router runs out of port numbers). You can read the whole NAT caveats article in the CT3 wiki.

Interesting links | 2008-07-27

Browsing through the starred articles in my FeedReader, I also found two older articles:

OSPF area default-cost is a 16-bit quantity

The area area default-cost number router configuration command changes the cost of the default route advertised into a stub or NSSA area. IOS documentation claims the cost is a 24-bit number (and both type-3 and type-5/7 LSAs have 24 bits available for the metric), but in reality the router accepts a 24-bit number and remembers the lower 16 bits, potentially resulting in quite unexpected behavior (just try setting the OSPF costs to 30000 on one ABR and 70000 on another).

Update (2008-08-08): This behavior is reported as bug CSCsl12946 and fixed in IOS release 12.4(20)T.

Here is a sample printout taken from a router running IOS release 12.4(15)T5; 12.2(33)SRC behaves identically.
A1(config)#router ospf 1
A1(config-router)#area 11 default-cost 65538

A1#show running | sect router ospf
router ospf 1
 log-adjacency-changes
 area 11 stub
 area 11 default-cost 2

A1#show ip ospf 1 database summary 0.0.0.0

            OSPF Router with ID (10.0.1.1) (Process ID 1)

                Summary Net Link States (Area 11)

  LS age: 16
  Options: (No TOS-capability, DC, Upward)
  LS Type: Summary Links(Network)
  Link State ID: 0.0.0.0 (summary Network Number)
  Advertising Router: A1
  LS Seq Number: 80000005
  Checksum: 0x4CE0
  Length: 28
  Network Mask: /0
        TOS: 0  Metric: 2 

Router configuration partitioning

If you have to troubleshoot routers with long configurations, you're probably as fed up with the slow response of the show running-config command as I am. Unfortunately, there's not much you can do; the running configuration is reverse-engineered from various memory variables every time you ask for it and that process simply takes time if you've configured many parameters.

IOS release 12.2(33)SRB has introduced a fantastic feature: router configuration partitioning. The early seeds of this idea are already present in mainstream IOS releases. For example, you can display the configuration of a single interface, all class-maps or all policy-maps. The configuration partitioning gives you the ability to display access-lists, route-maps, static routes, router configurations ...The following printout shows you the various parts of router configuration you can display:

PE-A#show running-config partition ?
  access-list All access-list configurations
  class-map All class-map configurations
  common All remaining unregistered configurations
  global-cdp All global cdp configurations
  interface Each Interface specific Configurations
  ip-as-path All IP as-path configurations
  ip-community All IP community list configurations
  ip-domain-list All ip domain list configurations
  ip-prefix-list All ip prefix-list configurations
  ip-static-routes All IP static configurations
  line All line mode configurations
  policy-map All policy-map configurations
  route-map All route-map configurations
  router All routing configurations
  snmp All SNMP configurations
  tacacs All TACACS configurations
For example, if you want to display just the configuration of the OSPF process, you'd ask for show running partition router ospf 1:
PE-A#show running partition router ospf 1
Building configuration...

Current configuration : 164 bytes
!
Configuration of Partition - router ospf 1
!
!
router ospf 1
 log-adjacency-changes
 passive-interface Serial1/1
 network 0.0.0.0 255.255.255.255 area 0
!
!
end

Unequal-bandwidth EBGP load balancing

EIGRP was always described as the only routing protocol that can do unequal-cost load sharing. As it turns out, BGP is another one (although it's way more limited than EIGRP). For example, if you have two links into a neighbor AS, you can load-share across them proportionally to their bandwidth. You can find all the details, sample configurations and router printouts in the CT3 wiki.

Goodbye fast switching & cell-mode MPLS

After leaving us in the dark for almost a year, Cisco finally released new functionality in IOS release 12.4(20)T. Support for a number of hardware platforms has been removed (dynamips fans are left with the 7200’s, everything else is gone). They also removed two switching features: fast switching and label-controlled ATM (cell-mode MPLS-over-ATM) together with Label Switch Controller (LSC).

I have no problem living without LC-ATM or LSC; this technology was primarily a retrofit for the old boxes by the time MPLS really took off with MPLS VPN. Fast switching is a different beast. Whenever you’d encounter bugs in more creative designs involving NAT, IPSec and GRE on low-end routers, you could turn off CEF (assuming you did not run NBAR) and things would (sometimes) miraculously start to work. Without fast switching, turning off CEF would bring you straight into process switching, resulting in an order-of-magnitude (or more) performance loss. On the other hand, it's obvious it makes no sense to maintain an obsolete switching method … and more bugs will probably get reported and fixed now that the easy escape route is gone.

Next-hop fixup in partially-meshed NBMA networks

Shahid has sent me an interesting observation: he was running RIP in partially-meshed Frame Relay network and wanted to use static host routes to fix next-hop problems, but somehow they didn't work as expected. I ran a series of scenarios testing RIP, OSPF, EIGRP and BGP and found that some of the old tricks don't work any more. Furthermore, it's almost impossible to run RIP over partially-meshed NBMA network, as the default RIP next-hop processing messes up your routing table. The details are explained in the Next-hop fixup in partially-meshed NBMA networks article in the CT3 wiki.

This article is part of You've asked for it series.

Summer schedule

I don't know about the rest of the world, but Europe is evidently in the middle of the summer holiday season (at least judged by weekend-long traffic jams on the highways). To help you handle the information overload when getting back from your vacation, I'll limit myself to three or four posts per week. OK, I might also turn off the routers for a few days and do a few more interesting things :)

Shorter display of OSPF database

Recently I had to explore the behavior of Cisco IOS OSPF implementation and had to inspect OSPF database on routers in various areas. If you're only interested in the contents of the database (not in low-level troubleshooting), variety of LSA fields (including LS Age, Options, Checksum, Length ...) are just cluttering the printout, so I fine-tuned the show filter to exclude all the non-relevant fields, ending with show ip ospf database parameters | exclude LS|Options|Check|Len|(MTID:[ 0-9]+$) (the MTID field appears in IOS release 12.2SRC).To make the command more useful, I've changed it into a short Tcl script (using steps from the post explaining how to execute complex CLI commands from Tcl) stored in flash:ospfdb.tcl

set cmd {show ip ospf database }
append cmd $argv
append cmd { | excl LS|Options|Check|Len|(MTID:[ 0-9]+$)}
puts [exec $cmd]
… and defined alias exec ospfdb flash:ospfdb.tcl. I could then easily inspect the contents of various parts of OSPF database I was interested in, for example:
a3#ospfdb external 0.0.0.0
 
            OSPF Router with ID (10.0.1.3) (Process ID 1)
 
                Type-5 AS External Link States
 
  Link State ID: 0.0.0.0 (External Network Number )
  Advertising Router: 10.0.1.5
  Network Mask: /0
        Metric Type: 2 (Larger than any link state path)
        Metric: 1 
        Forward Address: 0.0.0.0
        External Route Tag: 1
You can find more Tclsh-related information in the Tclsh on Cisco IOS tutorial. Sample Tclsh scripts are available in the Tclsh script library. If you need expert help in planning, developing or deploying Tclsh scripts in your network, contact the author.

Interesting links | 2008-07-13

How obscure can it get?: BGP IPv6 printouts

If you want to display any IPV6-related BGP objects (neighbors, routes …) you can use the familiar BGP commands, but have to prefix them with show ip bgp ipv6 unicast. For example, to display the BGP neighbors active in the IPv6 address family, you would use show ip bgp ipv6 unicast summary command. I doubt you like so much typing (I don't, just entering the IPv6 addresses is enough for me); luckily Cisco IOS has aliases - just configure alias exec bgpv6 show ip bgp ipv6 unicast and (for consistency) alias exec bgpv4 show ip bgp ipv4 unicast.

Update 2010-03-12: Cisco IOS also supports show bgp ipv6 unicast command, which (at least) makes BGP ipv4-agnostic. You'll find numerous IPv6-related BGP details as well as advanced backbone designs and configurations in the Building IPv6 Service Provider core workshop (register for the online webinar).

Global IPv6 strategies

If you want to understand the buzz raised recently about IP version 6, and your daily job includes more budget meetings, payroll discussions or strategy/operational planning than router configuration, Global IPv6 Strategies: From Business Analysis to Operational Planning (Cisco Press, 2008) is a mandatory book for you. The authors, Patrick Grossetete, Ciprian P. Popoviciu and Fred Wettling, are weathered veterans of the IPv6 battles, and their lengthy experience with IPv6 shines through the pages of this book.

QoS Policing in Cisco IOS

Policing implementations in Cisco IOS are a bit confusing: IOS supports three different algorithms that are configured with very similar parameters of the police command in modular QoS CLI. There's also the older rate-limit command that uses a limited implementation of one of the three algorithms. You'll find all policing details, including the graphic representations of all three algorithms in the QoS Policing in Cisco IOS article in CT3 wiki.

The value of being a CCIE

I was very pleasantly surprised by the supportive comments to my CCIE-related post; I didn’t realize there are so many CCIEs out there that feel the same way I do. Will we change anything? We can only hope; the CCIE program is orders of magnitude smaller than the Cisco’s equipment sales.

A few of the comments also asked for my opinion on the value of CCIE certification and whether it’s worth pursuing. Obviously, the short answer is yes.

CCIE certification has “commercial” as well as “academic” value. Undoubtedly, being a CCIE will (on average) increase your chances of getting a better-paid job. If you’re looking for jobs where the CCIE certification could help, you absolutely have to maintain the active status. For example, if you want to work for a Cisco’s partner, your CCIE status brings value only if you’re an active CCIE (suspended or inactive CCIEs don’t count toward the CCIE quota Cisco partners have to maintain). This requirement makes sense: partner CCIEs are usually faced with critical production problems in customer networks and thus have to hone their skills continuously.

Conclusion: if you’ve just got your CCIE certification, make sure you re-certify a few times; losing the active status would destroy the value you’ve been working so hard to create.

The “academic” value of the CCIE certification is also worth mentioning: it’s one of the few certifications that you simply cannot fake with the help of brain dumps. If you want to prove to yourself (and others) that you can reach the expert level in networking, go for CCIE or JNCIE (anyone aware of any other certifications that involve a day in an actual lab?). Of course, once you’ve become CCIE, it is worthless discussing whether you’re still a CCIE if you haven’t recertified (only certification zealots have problems with this concept). If you’ve got a PhD in physics, nobody will question whether you should still have the degree if you don’t work with particle accelerators; it’s the same with the CCIE certification (although Cisco is mysteriously vague on this topic).

Last but not least, the traffic logs have undoubtedly proved that I should stop writing about technology: the daily visits have jumped on July 1st and stayed higher-than-average for the next two days.

Wonderful Cisco IOS documentation

I wanted to figure out the exact release when Cisco IOS got EBGP load-sharing. Fighting through Feature Navigator is a pain for me, so I usually check older IOS documentation, starting from the old UniverCD URL (I was able to remember that one, the new URLs don't make any sense to me). When I've selected IOS release 11.0 configuration guides, the system redirected me to the "new style" 11.0 Router Products Configuration Guide ... and it looks like IOS did not support IP in release 11.0 :) ... or at least there are no instructions on how to configure it.

It's really sad how Cisco handles documentation these days. First they'd moved everything to new addresses and implemented redirects that didn't work (this is mostly fixed now), now they've managed to lose important parts of documentation.

Simple CLI extensions: handling special characters

Last week I've described how you can extend the exec-mode CLI commands with almost no knowledge of Tcl. A bit more work is required if your commands include Tcl special characters (quotes, braces or backslashes).

For example, to display all routes advertised by customers of AS X, you'd use the following show command: show ip bgp regexp _X_([0-9]+)(_\1)*$ (the regular expression is explained in the AS-path based filter of customer BGP routes post). This command cannot be entered as a Tcl string with variable substitution; Tcl would interpret the [ and \ characters. You could enter the whole command in curly braces, but then there would be no variable substitution that we need to insert command line parameters. To make Tcl happy, use the following Tcl commands:

  1. set cmd {first-part-of-command} stores the command prefix into the cmd variable;
  2. append cmd $argv appends the command line arguments to the command;
  3. append cmd {rest-of-command} appends the rest of the IOS exec command;
  4. puts [exec $cmd] executes the command and prints the results.

For example, the following code will display the customers of a BGP AS specified in the command line (after being stored in a flash file and defined in an alias, of course):

set cmd {show ip bgp regexp _}
append cmd $argv
append cmd {_([0-9]+)(_\1)*$}
puts [exec $cmd]

Interesting links | 2008-07-06

Wikis and blogs

A week ago I've announced that we've managed to merge my personal wiki efforts with NIL's corporate wiki, resulting in a solution that will cover more networking topics than just the Cisco IOS issues I've been describing so far. The only response I've got was a concerned anonymous reader who was afraid he would have to sort through a few more entries before getting to the Cisco IOS content he's interested in.

Anyhow, the comment clearly shows that I haven't done a good enough job detailing what my plans are (in case anyone wants to know :). Blogs (particulalry Blogger-based ones) are not the best media to write long in-depth texts. I've therefore decided a few months ago that I will have to start a Wiki along with the IOS hints blog to split the content based on its length and technical depth: the longer articles would be stored in the Wiki (also giving you the ability to expand on them or fix my errors), whereas the shorter hints would stay in the blog. Also (as you've probably noticed), whenever I publish a Wiki article, I'm also publishing a link and a short introductory text in the blog, so you'll be aware of all my content just by following the blog (or its RSS feed). The only change (as far as you are concerned) is that the Wiki will have a broader, but still networking-focused, coverage than if it would remain my personal effort.

Just for the reference, this is what I wrote when I was documenting the wiki-to-blog positioning for people I've invited as potential contributors:

The next obvious question should be: why not the blog. The reason is very simple: ease of use and the underlying assumptions. I've started blogging because it was the easiest platform to push quick ideas into the Internet ... and that's what blogs are good for. But unless you invest a lot of energy into customization (and I can't do much of that, as I'm using Blogger), blogs still have a sequential mentality (and Blogger's first page gets quite slow if you're writing long articles). Furthermore, formatting my stuff on Blogger the way I want it to appear is close to a nightmare; I'm much faster using Wiki markup. Last but definitely not least, the wiki software allows me to edit a single section, so I can really focus on the text that needs to be fixed; in Blogger, I have to spend a lot of time trying to find the exact text in the small editing window (but of course, that's my fault, I'm misusing Blogger for something it was never designed to do).
There is also another huge difference between a blog and a wiki: a blog post is almost static (only the author can fix errors in it), while a wiki is a dynamic environment. Hopefully we can attract writers as well as readers and have other people fix the typos, bugs or even add explanations where ours are sketchy.

In case you'd like to get more overview information on differences between blogs and wikis, check my Web 2.0 presentation.

The OSPF default mysteries

A while ago I wanted to combine the blog posts I've written about the default routes in OSPF into a single wiki article. As I started to investigate the various options you have to generate default routing in OSPF (including stub areas and not-so-stubby areas), the text quickly became too long and resulted in July IP Corner article "The OSPF default mysteries".

Anycast

Whenever I start digging into technical details, I learn something new. A few days ago I've stumbled across the term anycast, which is a very interesting way to solve scalability issues:

  1. Deploy geographically dispersed servers using the same IP address (obviously they would also need a unique IP address or you wouldn't be able to manage them);
  2. Advertise your service as residing on that IP address (for example, use the IP address in NS records for DNS zones you host)
  3. Advertise the IP address (or corresponding IP prefix) into the Internet from multiple locations.
Although you could (theoretically) use anycast technology for any application, it works best with simple request-response UDP applications (for example, DNS). In all other scenarios, the application session would fail if the nearest anycast server would change following a change in the network.

Multihomed EIGRP sites in MPLS VPN network

Deploying EIGRP as the PE-CE routing protocol in MPLS VPN networks is easy if all sites have a single PE-CE link and there are no backdoor links between the sites. Real life is never as simple as that; you have to cope with various (sometimes undocumented) network topologies. Even that would be manageable if the customer networks would have a clean addressing scheme that would allow good summarization (that happens once in a blue moon) or if the MPLS VPN core could announce the default route into the EIGRP sites (wishful thinking; the customer probably has one or more Internet exit points).

In the end, you’re left with two-way route redistribution between core MP-BGP and edge EIGRP, resulting in nightmarish scenarios (probably a good half of the blog posts of the CCIE candidates talk about redistribution horrors). Fortunately, Cisco implemented two extra features supporting EIGRP-to-MP-BGP redistribution: BGP cost community and BGP Site-of-Origin. The Multihomed MPLS VPN sites running EIGRP article I wrote in CT3 wiki describes how you can use both features to get a reliable network even when doing two-way redistribution.

Why I'm no longer an active CCIE

July 1st, 2008 marks another milestone in my professional career: I became an inactive CCIE. Before going into the details of why I decided not to go for the recerfitication exam (I haven't even tried to go there), let me just say that I've been working in the networking industry for 25 years and had the CCIE status for the last 13 years. I no longer see myself craving for jobs where the activity of my CCIE status would count and the "Benefits of CCIE Membership" (including the party at Cisco Live! event) are not coming close to giving me any motivation to extend the status.

However, the real reason I decided not to extend my active status lies in the process. Years ago, Cisco organized update trainings for CCIEs. Attending one of these trainings (which really added value to your knowledge) extended your status. In my opinion, an update training combined with a post-training exam would make sense. Like many other features of the program, these trainings are long gone.

Passing a written exam every two years with more-or-less the same questions is (in my personal opinion) bogus. It does not require me to grow or acquire new knowledge, it just forces me to re-read the IP multicast and IS-IS student kits we've developed. It's simply a tick in the box and I'm no longer willing to participate in this charade. To make matters worse, the tests were not exactly accurate over the years I had to take them. When I was developing (the then only) EIGRP training for internal Cisco audiences, I lost most points on EIGRP questions simply because I knew too much about the protocol. A few years ago I was faced with purely marketing questions about a newly-promoted technology that were obviously hastily added to the pool of questions. To be honest, I was told that the current test should be better that my past experiences, but I decided I will not find out how true that is. I had enough.