Build the Next-Generation Data Center
6 week online course starting in spring 2017

Small enhancement in zone-based firewalls

In the Deploying Zone-Based Firewalls book I wrote:

In early releases supporting zone-based policy firewall configuration (IOS 12.4(6)T), match protocol command cannot be used to classify traffic to or from the self zone. Only IP access lists can be used for traffic classification purposes.
Misha Volodko reported that the match protocol icmp command works for him when used with the self zone. Another small step toward perfect implementation :) ... and don't forget that you can always use class class-default to catch all the unclassified traffic (and log it before it's dropped, for example).

Interesting links | 2008-08-30

Load balancing with parallel EBGP sessions

Did you know you can run multiple parallel EBGP sessions between a pair of BGP-speaking routers? This capability gives you the most versatile EBGP load sharing design.

Read more in the CT3 wiki.

SNMP v3 users not shown in running-config

Ralf sent me a SNMPv3 question:

If I create a SNMPv3 user which has a password (snmp-server user userthree groupthree v3 auth md5 user3passwd), this user does not appear in the running- or startup-config. Cisco even documents this if you know what to look for.

I strongly suspect (although I did not test this) that these users are also missing from configuration exported to TFTP servers. What would be the recommended way to make usable config backups of routers with such users?
Like certificates, the SNMPv3 users are stored in private-config and thus never appear in the router configuration. If you want to have a backup of the user data, create a text file on one of your NMS servers, add SNMPv3 usernames and passwords in the text file and use the copy somewhere running-config to configure SNMPv3 users on the routers.
This article is part of You've asked for it series.

IOS scheduling parameters

Peter Weymann sent me a really intriguing question:

A few days ago I started reading the Ciscopress book End-to-End Network Security: Defense-in-Depth and stumbled over the scheduler command. This one could be used to allocate time that the cpu spends on fast switching packets or process switching packets, if I understand it correctly. They also mention interrupting CPU processes but honestly I don't really understand how it works.

Cisco routers support (at least) three forms of layer-3 switching (formerly known as routing). CEF switching and fast switching are performed entirely within the interrupt context (I/O adapter interrupts a process the CPU is currently executing and all the work is done before the process resumes). Process switching is performed in two steps: packet is briefly analysed within the interrupt context and requeued into the IP Input process where it's eventually switched. Almost all I/O adapters used these days use a concept of RX/TX rings to communicate with the CPU, meaning that the CPU potentially has to handle more than one packet for each interrupt.

Fast switching is gone starting with IOS release 12.4(20)T.

Under very high load, the packet arrival rate could be so high that the router would constantly service packets within the interrupt context without ever returning back to the IOS processes.

You can check the CPU load incurred by the interrupt context and IOS processes with the show process cpu command. The second number in the five seconds part of the first line tells you the amount of interrupt context activity in the last five seconds.

To prevent the starvation of IOS processes (which could result in keepalive and routing protocol problems, eventually leading to loss of routing protocol neighbors), the scheduler allocate command limits the amount of time that can be spent in the interrupt context and allocates some guaranteed time to the IOS processes. Very probably the routers have a mechanism to mask the requests from the I/O adapters during that period so that the CPU is not interrupted (BTW, this slightly increases the jitter).

A similar command is the scheduler interval command. IOS has high- and low priority processes. Whenever the CPU has to decide what process to run (usually following an interrupt or when a process decides it's done with its work), it will run a high-priority process if one is ready. This could lead to starvation of low-priority processes and the scheduler interval command specifies the maximum amount of time the higher-priority processes can consume before a low-priority process is given a chance to run.

Unless you have serious (and I mean __serious__) problems in your network, don't play with these commands. They are a last-resort things you can do if you're under very heavy load and still need access to the exec to reconfigure the router. In most cases, you should not have to worry ... and anyhow, if the CPU load is close to 100%, you have other problems anyway.

Apart from the Inside Cisco IOS Software Architecture book that you absolutely must have if you're interested in (a bit outdated) view of the internals of Cisco IOS, you can get more information in these documents:

RIP route database

Did you know that RIP, the venerable routing protocol that is present in Cisco routers for the last 20 years, uses an internal database, not the IP routing table, to process RIP updates? This database contains no fancy information (like EIGRP topology table) that would allow RIP to converge faster, but there are still minor differences between the RIP database and the IP routing table.

Read the full article in CT3 wiki …

Do you have a good reason to use BGP aggregation?

For the last 10 years, I've been preaching that you should use static BGP prefix advertising (with the network mask router configuration command) to advertise your IP address space into the public Internet, not the BGP aggregation. I might see some use for BGP aggregation in enterprise networks (or MPLS VPN networks) using BGP as the core routing protocol with other routing protocols serving the edge, but I cannot find a good scenario where BGP aggregation in public Internet would be a good solution. Do you use BGP aggregation in your network? Do you have a good scenario that you'd like to share with us? Write a comment.

New in CT3 wiki | 2008-08-24

Jeremy Stretch has generously contributed a few of his great articles into the CT3 wiki: the Policing versus Shaping article describes the performance impact both QoS mechanisms have on TCP sessions and the IPv6 EUI-64 article describes the algorithm that's used to transform a MAC address into an IPv6 address, resulting in automatic generation of unique IPv6 interface addresses.

Private IP addresses in public networks

Did you want to use private IP addresses in a public network? The short recommendation is: “don't”. If you use them in the network core, your customers might have problems with network troubleshooting; if you assign them to your customers and use NAT or PAT, you've just created serious security issues. You can find more details in the Avoiding private IP security risks in public networks article I wrote for SearchTelecom.com.

The list of all articles I wrote for SearchTelecom is available in the CT3 wiki.

Is a label imposed in case of Penultimate Hop Popping?

Shivlu Jain sent me an interesting question:

I'm wondering whether a router performing penultimate hop popping (PHP) imposes an IGP label or not.The value of implicit null is 3; does it mean the router imposes this label (and adds four bytes to the packet)?
The penultimate router does not impose the IGP label (that's why this behavior is called penultimate hop popping). However, the egress router has to signal to its upstream neighbor (the penultimate router) that it should NOT impose a label, so it uses "implicit null" label (= 3) in TDP/LDP updates to signal that the top label should be popped, not rewriten.
This article is part of You've asked for it series.

Reverse lookup of OSPF router IDs

If you store the reverse mapping for the routers’ loopback interfaces in DNS or configure the name-to-address mappings with the ip host commands, you can use the ip ospf name-lookup global configuration command to display the OSPF router IDs as router names.

For example, I’ve configured names of all provider routers’ loopback interfaces in one of my MPLS VPN labs:

P#show run | inc host
ip host PE-A 10.0.1.1
ip host PE-B 10.0.1.2
ip host PE-C 10.0.1.5
ip host P 10.0.1.6

After that, I’ve configured the ip ospf name-lookup command and the OSPF show commands started printing names instead of IP addresses:

P#show ip ospf database

            OSPF Router with ID (10.0.1.6) (Process ID 1)

                Router Link States (Area 0)

Link ID    ADV Router   Age   Seq#       Checksum Link count
10.0.1.1   PE-A         56    0x80000002 0x000C1F 3
10.0.1.2   PE-B         57    0x80000002 0x003FD8 3
10.0.1.5   PE-C         59    0x80000002 0x0027CF 3
10.0.1.6   P            53    0x80000003 0x00B675 7

Unfortunately, this functionality does not implement its full potential. For example, when examining router link states, the router ID is displayed as name, but the adjacent router IDs are not. Too bad, you still have to know the router IDs by heart .

P#show ip ospf database router 10.0.1.6

            OSPF Router with ID (10.0.1.6) (Process ID 1)

                Router Link States (Area 0)

  LS Type: Router Links
  Link State ID: 10.0.1.6
  Advertising Router: P
  LS Seq Number: 80000003

    Link connected to: another Router (point-to-point)
     (Link ID) Neighboring Router ID: 10.0.1.5
     (Link Data) Router Interface address: 10.0.7.30
      Number of MTID metrics: 0
       TOS 0 Metrics: 64

    Link connected to: another Router (point-to-point)
     (Link ID) Neighboring Router ID: 10.0.1.2
     (Link Data) Router Interface address: 10.0.7.18
      Number of MTID metrics: 0
       TOS 0 Metrics: 64

    Link connected to: another Router (point-to-point)
     (Link ID) Neighboring Router ID: 10.0.1.1
     (Link Data) Router Interface address: 10.0.7.10
      Number of MTID metrics: 0
       TOS 0 Metrics: 64 

Introduction to Embedded Menu Manager

One of the great new features introduced in IOS release 12.4(20)T is the Embedded Menu Manager (EMM), which allows you to build dynamic hierarchical menus with a combination of static definitions in XML and Tcl scripts. The Cisco documentation on this feature is quite cryptic, so I wrote an introduction to EMM, describing the major XML elements and the ways to use it.

Read more in CT3 wiki.

EEM violates RFC 822

Justin sent me an interesting problem description:

Emails that come from EEM don't have a body, it just puts anything after the body keyword in the subject.

I quickly wrote a simple EEM applet ...

event manager applet MailTest
 event none
 action 1 mail server "10.0.0.10" to "c@d" from "a@b" →
  subject "eeee" body "ffff"

... and started it with event manager run MailTest. The simple SMTP server I'm using for debugging EEM printed out the following message content:

Incoming mail ... from 10.0.0.1 received
From: <a@b>
To:   <c@d>

Date: Fri, 01 Mar 2002 00:01:34 +0000
Message-ID: <00002002000135003@R1>
From: a@b
To: c@d
Subject: eeee
ffff

As you can see, there is no empty line between the Subject header and the message body. A quick look in RFC 822 confirmed that there should be an empty line separating headers from the message body:

message =  fields *( CRLF *text )

Conclusion: EEM is not RFC822-compliant e-mail client.

Hint to EEM developers: if it works with sendmail, it's not necessarily correct.

This article is part of You've asked for it series.

Measure the cable lengths on a Catalyst switch

Ken McCoy sent me a short question:

At one point someone posted an article about a command you could run on the Catalyst switch that would give you back the distance of the cable between the switch and end device, but now I can't find it.
I remembered reading the same article and after I've figured out the underlying technology is called TDR (Time Domain Reflectometer), uncle Google immediately provided a reader tip from Csaba Farkas.
This article is part of You've asked for it series.

DNS view-groups don't work on subinterfaces

Working on an implementation of a split DNS design, I encountered an interesting bug in Cisco IOS: the ip dns view-group command works only on interfaces, but not on subinterfaces. As it’s a pure IP feature, there obviously no reason why it shouldn’t work on anything that has an IP address; obviously someone forgot to insert the correct entry in the parser tables.

Here’s a sample configuration session taken from a 7200-series router running IOS release 12.4(15)T5:

c7200(config)#ip dns view-list Test
c7200(cfg-dns-view-list)#interface Serial 1/0
c7200(config-if)#ip dns view-group Test
c7200(config-if)#encapsulation frame-relay
c7200(config-if)#interface Serial 1/0.1 point
c7200(config-subif)#ip dns view-group Test
                        ^
% Invalid input detected at '^' marker.

CCIE is devalued? Get real.

My favorite provocateur has dreamed up another sensational story ... and even has numbers to back it up. Reverse engineering the increase in reported number of CCIEs and taking in account the estimated number of seats in Cisco's labs worldwide, he concluded that the pass rate for CCIE R/S is currently at 35% whereas in the past the rumors claimed it was only around 10%. The conclusions in the story should not surprise you ... it must be the braindumps and the devaluing of the CCIE program. Of course it's the braindumps: people like Petr Lapukhov, Jeremy Stretch, Arden Packeer, Joe Harris and tens of others (including yours truly) are dumping the contents of their gray cell matter into blogs and wikis, creating astounding amount of information that we've never got from Cisco in the past.

The CCIE preparation programs also cover an enormous amount of scenarios and variations, giving you lots of material to practice (BTW, when I was teaching CCIE preparation bootcamps 15 years ago, the pass rate of my students was over 90% as I simply forced them to configure all the possible stupidities Cisco IOS could do at that time). The tests don't have to get any easier; the participants (if the calculations are correct) are simply better prepared. Whether the increased number of CCIEs results in the perceived devaluing of the program is another question (remember: the supply/demand rules), but I am absolutely sure that people passing CCIE lab exam these days know approximately as much as those passing it two or three years ago.

Of course you could argue whether someone who did tens (or sometimes hundreds) of scenarios in his lab and then passed the CCIE test is an expert or a braindump cheater (let's wait for the first blog post that claims that), but I doubt anyone is able to remember so many recipes and apply the correct one without a profound understanding of the underlying issues.

Primary/Backup Area Border Router designs

In the IP Corner article The OSPF default mysteries I've mentioned the primary/backup ABR design scenarios and provided configuration examples for stub and NSSA areas. For those of you who'd like to see more in-depth design scenarios, I've described a few of them in the Primary/backup area border router designs article in the CT3 wiki.

NIL has a new home page

In the last few days, our software development team has launched a completely redesigned NIL's home page, which makes it easier to find exactly what you need, be it remote labs, classroom training, product descriptions or information on our consulting and network design services.

Make the "show" command available in configuration mode

I tend to forget whether I'm in configuration mode or not and often type the do command in exec mode or the show command in configuration modes. With the alias functionality you can make the show command a native command in the configuration modes; just configure alias configure show do show.

The “only” drawback of this approach is that IOS has zillion different configuration modes and you have to define the alias in each one of them (you could do it just in the most common ones … or try to remember to type the do keyword first :).

Interesting links | 2008-08-10

It looks like some people forgot that July and August are supposed to be summer months when everyone is off to the nearest beach enjoying the sun (or maybe the global warning has caused unusually rainy weather in some parts of the globe). Leading the pack is Petr Lapukhov (a few more weeks like this and I'll start suspecting he's actually an pseudonym for a group of people like Tony Li was rumored to be a decade ago), who described Per-VLAN spanning tree (PVST) protocol, Multiple Spanning Trees Protocol (MSTP) and Dynamic Multipoint VPNs (DMVPN) (this one is so long I'll probably never find the willpower to read it)

DMVPN is also covered by Jeremy Stretch (I'm starting to wonder what's the root cause for the sudden fascination with this solution), who also provided a nice introduction to EUI-64 IPv6 addresses, a very practical view on shaping-versus-policing dilemma and simple step-by-step introduction to 802.1X.

As one would expect, Joe Harris and Arden Packeer are also ignoring the summer temptations. Joe provided an interesting link to the CCDE practical exam demo and Arden is continuing with his "OSPF over Frame Relay" saga (a few more installments and he'll be getting close to Jordan's Wheel of Time).

And last but not least: Tim Riegert sent me a link to a page full of TCP/IP and IMS Sequence Diagrams. The diagrams serve as a demonstration of EventStudio System Designer capabilities, but they are still good.

BGP route reflector details

BGP route reflectors have been supported in Cisco IOS well before I started to develop the first BGP course for Cisco more than a decade ago (one of the early ABCT course descriptions is still available online thanks to the Wayback machine). It’s a very simple feature, so I was pleasantly surprised when I started digging into it and discovered a few rarely known details. Read all about the BGP route reflector details in the article I wrote in the CT3 wiki.

SSH works without AAA

I was always under impression that you have to configure AAA (even if you have local passwords) if you want to use SSH on a Cisco router. Based on the comment made by shef I tried various options and found out that SSH works without AAA (at least in IOS releases 12.4 and 12.2SRC). In both cases, you can configure AAA authentication (using AAA servers or local passwords) or local username/password authentication (you can also use enhanced password security).

This is the minimum configuration needed to support inbound SSH sessions on a router (you might want to add transport input ssh to the line configuration if you want to disable telnet access to the router):
hostname name
!
ip domain-name name
!
crypto key generate rsa
!
! define local usernames, use passwords or secrets
!
username a password b
username x secret y
!
ip ssh version 2
!
line vty 0 4
 login local

Related posts in IOS hints blog

Summer is a great time to complete small tasks you've always wanted to do but never found the time to do them. Thanks to Jackbook, individual post pages on my blog now include the Related posts by category feature.

Designing site-to-site IPSec VPNs

Boštjan Šuštar, one of our security gurus, started a formidable journey: he will document all Cisco IOS IPSec design architecture options in a series of IP Corner articles. In the first article, he's describing the venerable technology available "forever" in Cisco IOS – the static and dynamic crypto-maps. I particularly like the introductory paragraph explaining the need for IP encryption:

Once upon a time, in the land of IP, there was a wide area network (WAN) providing connectivity between clients and servers, and all was well. Then, suddenly, bad things started to happen, and paranoia spread throughout the land. Firewalls grew around hamlets to protect them from the unknown beyond the realm of calm, but then packets were forced to travel thorough the dark forests of the WAN. There was a need to provide them with protection.

Identifying TACACS+ failure

I've got an interesting question from Colin a while ago:

I would like to generate a different prompt during the login to the router if the TACACS+ server has failed, indicating to the network operators that they have to log-in with the special (local) username, not with the TACACS+ authenticated username/password.

Fortunately he was running TACACS+ which supplies its own prompts during the authentication phase (the solution would not work with RADIUS). If you change the local authentication prompts, you'll get the prompts from TACACS+ server if it's reachable from the router (the AAA authentication is performed via TACACS+ server) and the local prompts if the TACACS+ server has failed (the AAA authentication is performed via any other mechanism). Here's a sample configuration:

aaa new-model
aaa authentication login REMOTE group tacacs+ local
!
aaa authentication fail-message #
Local authentication failed.
#
aaa authentication password-prompt "Enter local password:"
aaa authentication username-prompt "Enter local username:"
!
user a secret b
!
line vty 0 4
login authentication REMOTE

OSPF in a VRF requires a box-unique router ID

It's obvious why two routers in the same OSPF domain cannot have the same router ID. But requiring unique router IDs on OSPF processes running in different VRFs is probably too harsh, although it does prevent confusion if two VRFs ever get connected through a customer site. Anyhow, if you have overlapping IP addresses on loopback interfaces in different VRFs, OSPF process might not start.

Here's a short example: two VRFs have loopback interfaces with the same IP address. Perfectly legal setup from the MPLS/VPN perspective.

c7200#show ip vrf interfaces
Interface    IP-Address   VRF            Protocol
Lo1003       10.0.0.1     EIGRP_OSPF     up
Lo1001       10.0.0.1     OSPF_1         up

However, when you try to configure the second OSPF process, it fails to start as it cannot get a box-unique router ID. You have to enter a different router ID manually.

c7200#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
c7200(config)#router ospf 1 vrf OSPF_1
c7200(config-router)#network 0.0.0.0 255.255.255.255 area 0
c7200(config-router)#exit
c7200(config)#router ospf 3 vrf EIGRP_OSPF
%OSPF-4-NORTRID: OSPF process 3 cannot pick a router-id.
  Please configure manually or bring up an interface with an ip address.
c7200(config-router)#router-id 10.0.0.2