Building network automation solutions

A lot of times, I get questions like "how do I configure BGP on my 2800-series router". As so much has been written about this topic, I'd just like to give you a list of resources:

• Cisco has published excellent in-depth document as part of Cisco IOS documentation
• A lot of BGP technical documents can be found in BGP resource center on CCO, including a simple document on BGP multihoming.
• Someone has put together a comprehensive list of BGP-related books on Amazon.com
• ... out of those books, the Internet Routing Architectures is a must-have.
• If you'd like to know more about BGP, attending a Configuring BGP course might be a good idea (more so as passing the corresponding exam counts toward your CCIP certification).
• If you're a Cisco partner, you can get BGP-related remote labs free of charge:
1. Start Partner e-learning connection.
2. After landing on the main PEC page, click here.
• Everyone else can buy the same set of remote lab exercises from NIL Data Communications.

Note: this article is part of You've asked for it series.

Cisco IOS allows you to configure the ip default-gateway, but most often it looks like this setting is ignored. In fact, the default gateway is only used when an IOS device does not perform IP routing (acts like an IP host), for example, when you configure a Catalyst switch for layer-2 switching ... or when you disable IP routing on a router with no ip routing configuration command. In both cases, the show ip route command (or show ip redirects on some Catalyst switches) displays the default gateway and any ICMP redirects received from directly attached routers:

b2#show ip route
Default gateway is 192.168.0.5

Host      Gateway         Last Use    Total Uses  Interface
1.2.3.4   192.168.0.10        0:00            13  FastEthernet0/0

Disabling IP routing on a router makes perfect sense if you use it as a (reverse) terminal server or telnet-to-X.25 gateway.

In his latest Q&A post, Scott Morris mentioned an excellent Cisco article that describes routing tricks needed to implement sinkholes and remote blackholes in great details. Highly recommended reading.

IOS release 12.2(13)T (integrated in IOS release 12.3) has added the capability to redirect output of an IOS show command to a file. This feature uses Unix-style pipes (similar to the include, exclude and section keywords) and adds append, redirect and tee (redirect + print) keywords.

The show output can be redirected to a local filename (in flash, on usb token or even in NVRAM) or sent to a remote server (currently only FTP and TFTP servers are supported). For example, the show ip interface brief | redirect ftp://student:[email protected]/ifstatus command will store the current interface status to an FTP server.

Note: the append (or tee /append) operation only works on destinations that support the file append operation: class-C flash file systems, local disks, USB tokens and NVRAM.

I've always wanted a short summary display of DLCIs configured on my Frame Relay boxes (or whatever your favorite WAN technology is), but the only printout I would get from the router would be the lengthy show frame pvc printout. Fortunately, a judicious use of output filters can get you a summary printout from almost anything Cisco IOS produces.For example, I would like to see just the highlighted lines in my show frame pvc printout:

b2#show frame pvc

PVC Statistics for interface Serial0/0/0 (Frame Relay DTE)

              Active     Inactive      Deleted       Static
  Local          1            0            0            0
  Switched       0            0            0            0
  Unused         3            0            0            0

DLCI = 101, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Serial0/0/0.101

  input pkts 1003          output pkts 948          in bytes 107601
  out bytes 106438         dropped pkts 68          in pkts dropped 68
  out pkts dropped 0       out bytes dropped 0
... rest deleted ...

The first line I'm interested in contains the pattern for interface, the second one DLCI USAGE. My output filter would thus have to match any one of these patterns:

b2#show frame pvc ¦ include (for interface¦DLCI USAGE)
PVC Statistics for interface Serial0/0/0 (Frame Relay DTE)
DLCI = 101, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Serial0/0/0.101
DLCI = 201, DLCI USAGE = UNUSED, PVC STATUS = ACTIVE, INTERFACE = Serial0/0/0
DLCI = 302, DLCI USAGE = UNUSED, PVC STATUS = ACTIVE, INTERFACE = Serial0/0/0
DLCI = 401, DLCI USAGE = UNUSED, PVC STATUS = ACTIVE, INTERFACE = Serial0/0/0

And to add icing on the cake, I defined an alias with the alias exec dlci show frame pvc ¦ include (for interface ¦ DLCI USAGE) configuration command. Now I can display the FR DLCI status with a simple dlci command.

Sometimes, parts of router configuration get lost during the reload process: although the configuration commands are saved in NVRAM, they are not processed after the reload and thus do not appear in the running configuration. Re-entering these commands manually solves the problem ... but it's obviously not a reliable solution.

Embedded Event Manager (EEM) solves this issue as well. You just configure an applet that triggers on syslog message SYS-5-RESTART and reapplies the necessary configuration commands.For example, to fix the bug CSCsf32390, Cisco recomments the following applet:

event manager applet add-buffer
 event syslog occurs 1 pattern "%SYS-5-RESTART: System restarted"
action 1.0 cli command "enable"
action 2.0 cli command "configure terminal"
action 3.0 cli command "buffers particle-clone 16384"
action 4.0 cli command "buffers header 4096"
action 5.0 cli command "buffers fastswitching 8192"
action 6.0 syslog msg "Reinstated buffers command"

In my April IP Corner article, Scaling EIGRP Networks with Stub Routers, I've described how you can use EIGRP stub routers to improve the convergence time of large EIGRP networks and increase their stability. Now you can augment the article with a recording of the virtual classroom presentation I did a few days ago, which gives you even more in-depth details on the stub router technology and modifications Cisco made to EIGRP algorithms.

TACACS+ protocol introduced with the IOS AAA architecture had great provisions for customizing the whole login process (user-defined banners, prompts ...). Unfortunately, it never really took off and most AAA solutions deployed today rely on RADIUS servers that cannot control the login process itself (the RADIUS server can only check the username/password pair for validity).

To change the login prompts when using RADIUS servers, use the aaa authentication [banner|fail-message|password-prompt|username-prompt] text configuration command.For example, to introduce meaningful prompts when using one-time password solution, you could use something similar to this configuration:

aaa authentication banner #
Access to this router is protected with one-time passwords.

Send an e-mail to [email protected] if you need access.

#
aaa authentication fail-message #
Login failed. Wait at least 30 seconds and retry
#
aaa authentication password-prompt "Enter your PIN + one-time password:"
aaa authentication username-prompt "Enter your username:"

Note: the texts specified with the password-prompt and username-prompt options are one-line texts delimited with quotes, the parameters of the banner and fail-message options are multi-lined texts delimited with any character.

Earlier IOS versions treated changes in EIGRP summary address configuration (configured with the ip summary-address eigrp interface configuration command) very disruptively: all EIGRP sessions across the affected interface were cleared, sometimes resulting in a large number of routes entering active state, potentially leading to a stuck-in-active condition.

Recent IOS releases are more lenient: router with a change in summary address requests a resync (logged as graceful-restart on adjacent routers). A lot of updates and queries are still sent, but the adjacencies themselves are preserved:

• When configuring a summary route, all more specific prefixes on downstream routers enter active state.
• When a summary is removed, only the summary prefix itself enters active state and the affected router sends queries to all its neighbors, while the more specific prefixes are sent as regular EIGRP updates to the neighbors across the affected interface.

A change in EIGRP summary generates the following output on the router under configuration:

a1(config)#interface serial 0/0/0.100
a1(config-subif)#ip summary-address eigrp 1 0.0.0.0 0.0.0.0
%DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 172.16.1.2 (Serial0/0/0.100) is resync: summary configured
a1(config-subif)#no ip summary-address eigrp 1 0.0.0.0 0.0.0.0
%DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 172.16.1.2 (Serial0/0/0.100) is resync: summary configured

... and the downstream router generates log messages similar to these:

b1#
%DUAL-5-NBRCHANGE: IP-EIGRP(0) 1: Neighbor 172.16.1.1 (Serial0/0/0.100) is resync: peer graceful-restart

A lot of people arriving to my blog ask about continuous ping performed from a router. Well, you cannot generate never-ending ping from a command line interface, but you can get pretty close with a very large repeat count:

• Before starting the ping, set the line escape character to something you can generate from your keyboard (otherwise you won't be able to stop end the command). For example, terminal escape 3 will set the escape character to Ctrl-C.
• Start the ping with the ping ip destination repeat very-large-value command. For example, ping ip 10.0.0.1 repeat 1000000 will ping the target host longer than you'll be willing to wait.

Note: this article is part of You've asked for it series.

Sometimes you'd like to automate execution of command sequences or create a command alias that would trigger a series of commands. One way of achieving this is by creating an EEM applet. For example, to clear IP routing table and reset BGP neighbors, define the following EEM applet:

event manager applet ClearAll
 event none
 action 1.0 cli command "clear ip route *"
 action 2.0 cli command "clear ip bgp *"

You can trigger this applet with the event manager run ClearAll command or you could configure a command alias, for example alias exec cleanup event manager run ClearAll.

Note: this article is part of You've asked for it series.

We've just launched the e-learning version of the whole CCNP curriculum (the new version that was announced last autumn). As far as I'm aware (please correct me if I'm wrong :), this is one of the very few (if not the only) e-learning available for the new CCNP courses (ISCW and ONT).

Read more about this offering in the Blended Solutions Portfolio of NIL's web site.

I've almost started writing a Tcl procedure to display top-10 CPU-intensive processes on a router ... and then discovered the sorted option of the show processes cpu command. Even more, starting in IOS release 12.2T, the show processes cpu history command gives you a nice CPU utilization graph.Sample printouts are included below:

router#show processes cpu sorted 1min
CPU utilization for five seconds: 1%/0%; one minute: 2%; five minutes: 2%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
   5      180080      9762      18447  0.00%  1.75%  1.73%   0 Check heaps
  62         648       181       3580  0.00%  0.31%  0.12%   2 Virtual Exec
  25        4116       173      23791  0.49%  0.05%  0.00%   0 Per-minute Jobs
  30         848      1172        723  0.00%  0.01%  0.00%   0 IP Input
  81          12       357         33  0.08%  0.00%  0.00%   0 CEF Scanner
   6           8         2       4000  0.00%  0.00%  0.00%   0 Pool Manager
   4           0        86          0  0.00%  0.00%  0.00%   0 DHCPD Timer
   3           4        27        148  0.00%  0.00%  0.00%   0 CRYPTO IKMP IPC
   9           0         1          0  0.00%  0.00%  0.00%   0 AAA high-capacit
  10          52       238        218  0.00%  0.00%  0.00%   0 ARP Input
... rest deleted ...
router#show processes cpu history
                22222
    22          11111          11111
100
 90
 80
 70
 60
 50
 40
 30
 20             *****
 10             *****
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5
               CPU% per second (last 60 seconds)

    2121121112121112121 11111222222122 12211121119112121 12221
    1926405121716641818 76211100148411 70088401221831611470011
100
 90                                              *
 80                                              *
 70                                              *
 60                                              *
 50                                              *
 40                                              *
 30                               *              *
 20 **** **  ***** **** **   ****** ** ***** *   ** ***  ****
 10 ******************* **********#*** **********#****** *****
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%
     1
    80
    60
100  *
 90 **
 80 **
 70 **
 60 **
 50 **
 40 **
 30 **
 20 **
 10 **
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5
                   CPU% per hour (last 72 hours)
                  * = maximum CPU%   # = average CPU%

The Warm Reload functionality introduced in IOS release 12.3(2)T significantly reduces the reload time. In my test lab, the reload time of a Cisco 2800 router booting from flash was reduced from 135 to 54 seconds as measured by the %SYS-6-BOOTTIME: Time taken to reboot after reload ... syslog message.

The theory behind warm reload is simple: the router saves initial data (as stored in IOS image) in a separate memory region and reuses saved data together with IOS code already residing in RAM to restart IOS. Of course, the IOS code (depending on platform's memory management capabilities) or saved data could get corrupted, therefore the warm reload cannot be used continuously (and the router falls back to traditional reload if the router crashes before a specified time interval).

Warm reload is configured with the warm-reboot count number uptime minutes configuration commands. After it has been configured, a router reload (or power-up) is needed to initialize the saved data region. When the warm reboot is operational (as verified with the show warm-reboot command), you can use reload warm command to start it.The output of the show warm-reboot command displays all the relevant setup parameters as well as the amount of memory used by this feature:

a2#show warm-reboot
Warm Reboot is enabled
Maximum warm reboot count is 5
Uptime after which warm reboot is safe in case of a crash is 2 (min)

Statistics:

0 warm reboots due to crashes and 0 warm reboots due to requests 
have taken place since the last cold reboot
2823 KB taken up by warm reboot storage

The saved data region is also displayed with the show region command:

a2#show region
Region Manager:

      Start         End     Size(b)  Class  Media  Name
 0x0F400000  0x0FFFFFFF    12582912  Iomem  R/W    iomem:(uncached_iomem_region)
 0x3F400000  0x3FFFFFFF    12582912  Iomem  R/W    iomem
 0x40000000  0x4F3FFFFF   255852544  Local  R/W    main
 0x4000F000  0x431DFFFF    52236288  IText  R/O    main:text
 0x431E0000  0x45F8C25F    47891040  IData  R/W    main:data
 0x45F8C260  0x465FFA5F     6764544  IBss   R/W    main:bss
 0x465FFA60  0x468C19AF     2891600  Local  R/W    main:saved-data
 0x468C19B0  0x4F3FFFFF   146007632  Local  R/W    main:heap
 0x80000000  0x8F3FFFFF   255852544  Local  R/W    main:(main_k0)
 0xA0000000  0xAF3FFFFF   255852544  Local  R/W    main:(main_k1)

When pinging a directly-attached host (end-station) from a router, it's quite common to lose the first reply, as shown in the following example (the same symptom might occur when pinging a remote host that has been inactive).

a2#ping 10.0.0.10

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.0.10, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/4 ms

Actually, it's not the reply that was lost, the request was never sent out. Whenever a router has to send a packet to the next-hop (or directly attached destination) that has no entry in the ARP table, the ARP request is sent out, but the original packet is unconditionally dropped.

In a previous post, I've described how to execute a Tcl file with the tclsh command. You can do even more than that: you can pass parameters to the executed file. Every word you enter after the file name in the tclsh command line is passed as a parameter to the Tcl code you execute. To get these parameters in Tcl, use Tcl commands similar to the code below:

# loop.tcl: changes loopback state
#
# syntax: tclsh loop.tcl ifnum state
#
set ifnum [lindex $argv 0] # first parameter after file name
set ifstate [lindex $argv 1] # second parameter after file name
if {[string equal $ifstate ""]} {
return -code error "Syntax: loop.tcl ifnum ifstate"
}
... rest of procedure ...

You'll find more details in the Tclsh command line parameters article in the CT3 wiki.

The April IP Corner article describes EIGRP stub routers, a feature that can significantly improve the stability of large EIGRP networks. I'm describing the theory behind EIGRP stub routers, how they reduce EIGRP traffic and decrease the convergence times in large hub-and-spoke networks as well as how you use undocumented IOS features to build fully redundant dual router stub sites.

One of my readers has asked an interesting question: can you reload a router when pinging a specific IP address from it fails? While there are other ways of dealing with stuck interfaces or routing processes, sometimes such a drastic measure is the only workaround, so here's how you do it:

• Configure an IP SLA measurement (you'll find the necessary commands in the Not-so-Very-Static Routes section of my IP Corner article Small Site Multihoming). You might want to use the after parameter in the ip sla schedule command to ensure the router does not get reloaded immediately after the startup due to IP routing table not being populated.
• Configure a tracked object based on the IP SLA measurement with the track object-id rtr sla-id reachability command
• Configure an EEM applet that will reload the router if the tracked object enters the down state

Use configuration similar to the one below for the EEM applet:

event manager applet PingHasFailed
 event track 100 state down
 action 1.0 syslog msg "Ping has failed, reloading the router"
 action 2.0 reload

Note: this article is part of You've asked for it series.

By default, IOS routers use enable passwords to authenticate incoming HTTP (web) requests. You could also use local usernames as the authentication mechanism, or you could deploy full-blown AAA-based solution.To enable AAA-based HTTP authentication, you have to define two AAA lists (authentication login and authorization exec) and bind them to the HTTP server with the ip http authentication aaa command. A working configuration example is included below (this one uses AAA-based local username authentication, but of course you can replace that with RADIUS- or TACACS-based one).

aaa authentication login web local
aaa authorization exec web local
!
ip http authentication aaa login-authentication web
ip http authentication aaa exec-authorization web

Warning: the router executes AAA authentication/authorization for every HTTP request sent by the browser. It's thus very hard to integrate this solution with one-time passwords (unless you can cache the credentials on the AAA server).

If you would like to boot an ISR router (for example, a 2800) from USB flash, but cannot upgrade the ROMMON, all is not lost - you can specify the USB-based image with the boot system configuration command (for example, boot system usbflash1:c2800nm-advipservicesk9-mz.124-11.T1.bin), but the boot process will take significantly longer (this also applies to any other scenarios where the ROMMON cannot get the image specified in the boot system command):

• When the router is reloaded, ROMMON reads the router configuration and tries to start the required image
• If ROMMON cannot load the image specified in the boot system command, it starts the default image (usually whatever is stored in on-board flash)
• When the first IOS image loads (after being copied and decompressed in most cases), it discovers that it's not the correct image
• The now-operational IOS image loads the new image in RAM (in our case from usbflash1:), decompresses it and transfers the control to it

The Tcl procedures used to execute IOS commands in Embedded Event Manager (cli_open, cli_write ...) don't work when you start Tcl shell from command line interface. To execute IOS commands in this context, use:

• exec command to execute an exec-level command, for example exec "show ip route"
• ios_config mode command to configure the router

If the first parameter of the ios_config command is a global configuration command, you shall omit the second parameter (for example, ios_config "hostname router"). To configure a parameter in one of the sub-configuration modes (for example, interface state), use the first parameter to specify the configuration mode and the second parameter as the actual configuration command (for example, ios_config "interface loop 0" "no shutdown"). And if you want to configure BGP neighbor in VRF address family within the BGP routing process ... then you're out of luck, you can't do it with Tcl shell you just keep adding parameters (undocumented!) to the ios_config command. You can even perform several configuration tasks in one go, for example ios_config "interface loopback 111" "ip address 1.2.3.4 255.255.255.255" "description Test" "ip ospf cost 20".

Note: I knew what I wrote initially did not feel right (although that's what the docs say), so I simply had to go back and so some more testing.

An in-depth version of this article is available in the CT3 wiki