Build the Next-Generation Data Center
6 week online course starting in spring 2017

The versatile more command

With IOS file system (IFS) introduced in IOS release 11.3AA (integrated in 12.0), IOS got the more command that displays any local or remote file that you could specify with IFS. The obvious use of the more command would be display of startup configuration (more nvram:startup-config), but you could also display built-in Tcl EEM policies (for example, more system:lib/tcl/http.tcl) or remote router configurations (for example, more tftp://host/cfg-file). But that's not all, you could even troubleshoot web servers and display HTML generated by the web server (for example, more http://192.168.0.2/index.html).

Note: IOS documentation claims that the show running command is obsolete and that you should use more system:running-config. This is not true, as the show running command has a number of interesting options that are not implemented with the more command.

Filter sections of your running configuration

The IOS command line interface has long included unix-style pipes that you could use to limit the output generated by the show commmands. Initially, the only available filters were begin (include everything after the first regular expression match), end (stop the output at the RE match) or include (include only matching lines).

IOS release 12.3(2)T (integrated in 12.4) brought us a few new filters, among them the section filter that includes or excludes whole sections (start of section being defined by a line with no leading space). You can use this filter to focus on parts of your router configuration. For example, to display only the routing protocols configuration, use show running | section include router command.

Of course, you can go a step further and define an alias, for example alias exec events show running | include ^event manager configuration command defines the exec-mode events command that lists all EEM applets.

Store your EEM Tcl policies in NVRAM

Embedded Event Manager is a bit picky about the location of the EEM Tcl policies: although they are loaded into RAM when registered, they have to reside on the router itself. If you have a low-end router with no flash disk (I'm using 2800-series routers) or USB flash and you don't want to mess with your flash: device (to prevent accidental erasure), the only other place left is NVRAM:. Surprisingly, it works.First you define the directory where your EEM policies will be stored with the event manager directory user policy nvram:/ configuration command. Then you use the copy command to transfer Tcl file into NVRAM (for example, copy tftp://10.0.0.10/tcl/ep.tcl nvram:ep.tcl). Now you can register the policy with the event manager policy name configuration command.

To clean the NVRAM after you're done, use erase /all nvram: command, otherwise your files will be left there (even after write erase).

Simulate interface counters with QoS policy-map

If you need interface counters on subinterfaces or virtual interfaces, you can emulate them with an empty policy-map, for example:

policy-map Count
class class-default
!
interface Serial0/0/0.100 point-to-point
service-policy input Count
service-policy output Count
The service policy counters are then inspected with the show policy-map interface name command:
a1#show policy-map interface Serial 0/0/0.100

Serial0/0/0.100

Service-policy input: Count

Class-map: class-default (match-any)
10 packets, 840 bytes
1 minute offered rate 0 bps, drop rate 0 bps
Match: any

Service-policy output: Count

Class-map: class-default (match-any)
61 packets, 7084 bytes
1 minute offered rate 0 bps, drop rate 0 bps
Match: any

Minimum SDM configuration

Recently I had to investigate whether Cisco Router and Security Device Manager (SDM) uses a default username/password, so I was forced to figure out how exactly the SDM works. The minimum functionality needed on an SDM-enabled router is preconfigured Ethernet interface, HTTP server and (optionally) DHCP pool.Just in case you need to reset a router configuration into a state where you can reinitialize it with SDM, here's the minimum configuration to make it work:

! Use whatever your first Ethernet interface is here
!
interface FastEthernet0/0
ip address 10.10.10.1 255.255.255.248
duplex auto

ip http server
no ip http secure-server

ip dhcp pool SDM
network 10.10.10.0 255.255.255.248
Note: routers shipped from Cisco would usually include username cisco password cisco configuration command.

Upgrade 2800 series router to support USB boot

A while ago we were faced with a 2800-series router with no software in its CF card. As all 2800-series routers have USB slots, I started investigating whether you could boot the IOS image from an USB flash token. The product documentation states you can't, but as always, reality changes quicker than documentation in Ciscosphere.

To support IOS boot from USB token, you need a newer version of 2800-series ROMMON which you'll get from the ROMMON download page (registered CCO users only). You need ROMMON release 12.4(13r)T (description: C2800 ROMMON Upgrade; adds boot from usb flash drive capability).

Not all static routes were created equal

In his excellent question, Matt reminded me of (almost prehistoric) days when static routes pointing to a connected interface (not IP next-hop) had administrative distance zero. I also remembered that we've had interesting routing problems as those static routes actually behaved like connected routes (and were redistributed into routing protocols with redistribute connected command).

Reliable static routing

IOS release 12.3T introduced an interesting concept: static routes that are used only if an object a router is tracking (for example, next-hop router) is available. Named reliable static routing, this feature allows you to constantly ping next-hop router (or any other IP address) and use the static routes only if the tested destination is reachable.

While the reliable static routes are easy to understand and configure, a careless implementation can lead to interesting routing loops or other instability problems. You'll find the needed in-depth design and implementation guidelines in my IP Corner article Small Site Multi-homing (which is one of the prime applications for reliable static routing) in sections Not-so-Very-Static-Routes and End-to-End Connectivity Test. You can also download a virtual classroom recording where I'm discussing even more static routing caveats and ways around them.

Embedded Event Manager scripting center

You can find a variety of sample EEM scripts on Cisco forums. Quite a few of them are really interesting, illustrating various aspects of using TCL to program EEM (as well as what can be done with EEM to make your life simpler).

Frame Relay local switching

Cisco IOS supported Frame Relay switching (emulation of a Frame Relay switch) for a very long time. First they've implemented local switching, then remote switching over a GRE tunnel. With the introduction of generic Layer 2 transport across a layer-3 backbone (L2TPv3 or AToM), Frame Relay switching got integrated into the new infrastructure, but never implemented completely ... that is, until release 12.0(27)S and 12.4(11)T which finally supports local switching in the new architecture. We've also got a few extra goodies: now you can do DTE-to-DTE switching (interconnecting two Frame Relay switches with a router) or same-port switching (switching two DLCIs terminating on the same router port).The new configuration commands differ only slightly from the old ones. To start with, you have to define DLCIs with the frame-relay interface-dlci switched interface configuration command and use the connect global configuration command to establish switching between two DLCIs. A working example is included below:

frame-relay switching
!
interface Serial0/0/0
frame-relay intf-type dce
frame-relay interface-dlci 100 switched
!
interface Serial0/0/1
frame-relay interface-dlci 200 switched
!
connect FR100_200 Serial0/0/0 100 Serial0/0/1 200

Unequal load split with static routes

Unequal load-sharing with static routes is almost impossible as there is no configuration command to assign non-default traffic share count to a static route. For example, if you configure two default routes, one pointing to a low-speed interface and another one pointing to a high-speed interface, there is no mechanism to force majority of the traffic onto the high-speed link (IOS ignores interface bandwidth when calculating load sharing ratios).

You can, howerer, use a workaround: if you configure multiple routes for the same prefix pointing to the same interface, that interface will attract proportionally more outbound traffic.

For example, let's assume you have two point-to-point serial subinterfaces, one three times as fast as the other:

interface Serial0/0/0.100 point-to-point
bandwidth 1000
ip address 172.16.1.1 255.255.255.252
!
interface Serial0/0/0.200 point-to-point
ip address 172.16.1.5 255.255.255.252
bandwidth 3000

To shift more traffic onto Serial0/0/0.200, you can create two default routes pointing to the second interface, one pointing to the interface itself, the other one to the next-hop router:

ip route 0.0.0.0 0.0.0.0 Serial0/0/0.100
ip route 0.0.0.0 0.0.0.0 Serial0/0/0.200
ip route 0.0.0.0 0.0.0.0 172.16.1.6

This setup will give you a 1:2 sharing ratio. To shift even more traffic to the higher-speed interface, one has to get more creative

  • Create a bogus host route for a bogus next-hop pointing to the actual next-hop router (and make sure you don't advertise the bogus route into your routing protocols).
  • Configure yet another static route pointing to the bogus next-hop. Due to recursive lookup done by Cisco IOS, the bogus next-hop will be resolved into the actual next-hop IP address.

In our example, you could use:

ip route 10.255.255.1 255.255.255.255 172.16.1.6
ip route 0.0.0.0 0.0.0.0 10.255.255.1

The results are as expected: the traffic split is the desired 1:3 ratio

a1#show ip route 0.0.0.0 0.0.0.0
Routing entry for 0.0.0.0 0.0.0.0, supernet
Known via "static", distance 1, metric 0 (connected), candidate default path
Routing Descriptor Blocks:
172.16.1.6
Route metric is 0, traffic share count is 1
10.255.255.1
Route metric is 0, traffic share count is 1
* directly connected, via Serial0/0/0.100
Route metric is 0, traffic share count is 1
directly connected, via Serial0/0/0.200
Route metric is 0, traffic share count is 1

a1#show ip cef 0.0.0.0 0.0.0.0 internal
0.0.0.0/0, version 43, epoch 0, attached, per-destination sharing
0 packets, 0 bytes
via 172.16.1.6, 0 dependencies, recursive
traffic share 1
valid adjacency
via 10.255.255.1, 0 dependencies, recursive
traffic share 1
valid adjacency
via Serial0/0/0.100, 0 dependencies
traffic share 1
valid adjacency
via Serial0/0/0.200, 0 dependencies
traffic share 1
valid adjacency

0 packets, 0 bytes switched through the prefix
tmstats: external 0 packets, 0 bytes
internal 0 packets, 0 bytes
Load distribution: 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 (refcount 1)

Hash OK Interface Address Packets
1 Y Serial0/0/0.200 point2point 0
2 Y Serial0/0/0.200 point2point 0
3 Y Serial0/0/0.100 point2point 0
4 Y Serial0/0/0.200 point2point 0
5 Y Serial0/0/0.200 point2point 0
6 Y Serial0/0/0.200 point2point 0
7 Y Serial0/0/0.100 point2point 0
8 Y Serial0/0/0.200 point2point 0
9 Y Serial0/0/0.200 point2point 0
10 Y Serial0/0/0.200 point2point 0
11 Y Serial0/0/0.100 point2point 0
12 Y Serial0/0/0.200 point2point 0
13 Y Serial0/0/0.200 point2point 0
14 Y Serial0/0/0.200 point2point 0
15 Y Serial0/0/0.100 point2point 0
16 Y Serial0/0/0.200 point2point 0

Note: this article is part of You've asked for it series.

Looking for interesting Embedded Event Manager challenges

I've started collecting interesting EEM solutions, for example:

If you have an interesting challenge that you think might be solvable with EEM, please let me know (use the Send a message to Ivan link in my bio page) ... after reading the following smallprint:

  • I might not be able to respond to all your mail requests.
  • By sending me your problem, you're agreeing that I'm free to post its solution together with a (heavily reworded) description of your problem in this blog as well as any other publication I might decide to author.

Log IP SLA failures

The IP Service Level Agreement (SLA) measurement feature of Cisco IOS is extremely SNMP-oriented. For example, there is no easy way to generate a logging message when an SLA operation fails (or exceeds the threshold). However, Embedded Event Manager reading SNMP variables can help you get there.Before checking the status of an SLA operation, you need to schedule it (remember the SLA entry#, you'll need it when forming the SNMP object ID). For example:

ip sla 42
icmp-echo 172.16.1.2 source-interface Serial0/0/0.100
timeout 500
threshold 100
frequency 1
ip sla schedule 42 life forever start-time now
When the SLA measurement has been defined, create an EEM applet to monitor the status of the rttMonCtrlOperTimeoutOccured variable in the CISCO-RTT-MON MIB (note that the SLA entry# is the last number in the OID). The EEM applet triggers on value 1 (true: timeout) and re-arms on value 2 (false: no timeout).
event manager applet SLA_Timeout
event snmp oid 1.3.6.1.4.1.9.9.42.1.2.9.1.6.42 get-type exact entry-op eq entry-val 1 exit-op eq exit-val 2 poll-interval 5
action 1.0 syslog msg "Ping to 172.16.1.2 timeout"
Similarly, you can write an applet that triggers on SLA response time being over threshold by monitoring SNMP OID 1.3.6.1.4.1.9.9.42.1.2.9.1.7.sla# (the rttMonCtrlOperOverThresholdOccurred variable).
Note: this article is part of You've asked for it series.

Reporting a state of an IP route

Have you ever wanted to be notified when your router loses a specific route (for example, the default route toward the Internet)? In IOS release 12.4T you can do it with a combination of Enhanced Object Tracking and Embedded Event Manager 2.2.As the first step, you configure a track object that will monitor the state of an IP route with the track ip route command.

track 100 ip route 0.0.0.0 0.0.0.0 reachability
After the track object has been created, you can monitor its state with an EEM applet (in the configuration example it only generates a syslog message, but you could also send an email to the network administrator).
event manager applet TrackRoute_0.0.0.0
event track 100 state any
action 1.0 syslog msg "route is $_track_state"
And now a test run ...
a1#conf t
Enter configuration commands, one per line. End with CNTL/Z.
a1(config)#ip route 0.0.0.0 0.0.0.0 loopback 0
a1(config)#^Z
01:08:47: %SYS-5-CONFIG_I: Configured from console by console
01:08:51: %HA_EM-6-LOG: TrackRoute_0.0.0.0: route is up
a1#conf t
Enter configuration commands, one per line. End with CNTL/Z.
a1(config)#interface loopback 0
a1(config-if)#shutdown
01:09:10: %SYS-5-CONFIG_I: Configured from console by console
01:09:11: %LINK-5-CHANGED: Interface Loopback0, changed state to administratively down
01:09:21: %HA_EM-6-LOG: TrackRoute_0.0.0.0: route is down
As you can see from the example, there is a significant delay before the track object responds to a route change. You can adjust this delay with the track timer ip route seconds global configuration command (default is 15 seconds).

Unequal cost load-sharing

One of the most commonly asked load-sharing-related questions is "can I load-share traffic across unequal-cost links". In general, the answer is no. In order to load-share the traffic, you need more than one path to the destination and the only way to get multiple routes toward a destination in the IP routing table is to make them equal-cost (the only notable exception being EIGRP that supports unequal-cost load-sharing with the variance parameter).

There are, however, two cases where you can force unequal traffic split across equal-cost paths toward a destination: when using inter-AS BGP with the link bandwidth parameter and when using unequal-bandwidth traffic-engineering tunnels.

Note: You can read more about load sharing with MPLS TE in my IP Corner article Perfect Load-Balancing: How Close Can You Get?Due to the way MPLS TE autoroute is implemented in Cisco IOS, all tunnels toward the same destination appear as equal-cost paths, even when their TE bandwidths are not the same. For example, using a simple TE configuration ...

interface Tunnel0
ip unnumbered Loopback0
tunnel destination 172.16.0.21
tunnel mode mpls traffic-eng
tunnel mpls traffic-eng autoroute announce
tunnel mpls traffic-eng priority 7 7
tunnel mpls traffic-eng bandwidth 300
tunnel mpls traffic-eng path-option 1 explicit identifier 1
no routing dynamic
!
interface Tunnel1
ip unnumbered Loopback0
tunnel destination 172.16.0.21
tunnel mode mpls traffic-eng
tunnel mpls traffic-eng autoroute announce
tunnel mpls traffic-eng priority 7 7
tunnel mpls traffic-eng bandwidth 500
tunnel mpls traffic-eng path-option 1 explicit identifier 2
no routing dynamic
... you get two equal-cost paths in your IP routing table even though the tunnel mpls traffic-eng bandwidths are different:
a1#show ip route ospf
172.16.0.0 255.255.0.0 is variably subnetted, 6 subnets, 2 masks
O 172.16.0.21 255.255.255.255 [110/51] via 0.0.0.0, 00:11:06, Tunnel0
[110/51] via 0.0.0.0, 00:11:06, Tunnel1
O 172.16.0.22 255.255.255.255 [110/52] via 0.0.0.0, 00:11:06, Tunnel0
[110/52] via 0.0.0.0, 00:11:06, Tunnel1
When transferring the IP routing table into the CEF table, the router takes MPLS TE bandwidth in consideration, resulting in unequal traffic split proportional to the MPLS TE bandwidth:
a1#show ip cef 172.16.0.21 internal
172.16.0.21/32, version 55, epoch 1, per-destination sharing
0 packets, 0 bytes
tag information set
local tag: tunnel-head
via 0.0.0.0, Tunnel0, 0 dependencies
traffic share 3
next hop 0.0.0.0, Tunnel0
valid adjacency
tag rewrite with Tu0, point2point, tags imposed: {}
via 0.0.0.0, Tunnel1, 0 dependencies
traffic share 5
next hop 0.0.0.0, Tunnel1
valid adjacency
tag rewrite with Tu1, point2point, tags imposed: {}

0 packets, 0 bytes switched through the prefix
tmstats: external 0 packets, 0 bytes
internal 0 packets, 0 bytes
Load distribution: 0 1 0 1 0 1 0 1 0 1 0 1 1 1 1 1 (refcount 1)

Hash OK Interface Address Packets Tags imposed
1 Y Tunnel0 point2point 0 {}
2 Y Tunnel1 point2point 0 {}
3 Y Tunnel0 point2point 0 {}
4 Y Tunnel1 point2point 0 {}
5 Y Tunnel0 point2point 0 {}
6 Y Tunnel1 point2point 0 {}
7 Y Tunnel0 point2point 0 {}
8 Y Tunnel1 point2point 0 {}
9 Y Tunnel0 point2point 0 {}
10 Y Tunnel1 point2point 0 {}
11 Y Tunnel0 point2point 0 {}
12 Y Tunnel1 point2point 0 {}
13 Y Tunnel1 point2point 0 {}
14 Y Tunnel1 point2point 0 {}
15 Y Tunnel1 point2point 0 {}
16 Y Tunnel1 point2point 0 {}
Note: this article is part of You've asked for it series.

Weird VTY numbers in IOS release 12.4T

Someone managed to scramble the VTY numbers displayed with the show users and show line commands in IOS release 12.4. The VTY numbers in the show printouts no longer correspond to the numbers in the router configuration.For example, using the "standard" configuration ...

line vty 0 4
password test
logging synchronous
login
... the show line command displays VTY numbers 322 to 326 on a 2800 router:
a1#show line
Tty Line Typ Tx/Rx A Modem Roty AccO AccI Uses
* 0 0 CTY - - - - - 0
1 1 AUX 9600/9600 - - - - - 0
322 322 VTY - - - - - 0
323 323 VTY - - - - - 0
324 324 VTY - - - - - 0
325 325 VTY - - - - - 0
326 326 VTY - - - - - 0
Similarly, the show users command displays the same VTY numbers:
a1#show users
Line User Host(s) Idle
* 0 con 0 idle 00:00:00
vty 322 idle never
vty 323 idle never
However, the clear line command does not like the numbers provided in the show users printout.
a1#clear line vty 322
% 322 is an illegal line number [OK]
a1#clear line vty 0
[confirm]
[OK]

What is the sl_def_acl access list

Recenty, a lot of people were looking for information on the sl_def_acl access list. Here's the whole story: if you've configured IOS login enhancements on your router, the router generates an access list named sl_def_acl (unless you specify your own with the login quiet-mode access-class command) the first time it has to enter the quiet mode. This access-list is then applied to the VTY lines whenever the router enters the quiet mode and removed from the after the quiet period is over. The access list itself is left in the running configuration.

Default action in firewall policy maps

Marko asked a very interesting question: What is the default class policy in a firewall policy-map (policy-map type inspect)? Or, using his original wording, "is it mandatory to use class class-default/drop" at the end of every policy map?

As it turns out, the default action for any class (unless you specify otherwise) is drop. By default, packets not matched by any traffic class are therefore dropped (unless you specify a different action in the class-default), similar to well-known ip access-list behavior.I've tested this feature using a reverse setup: web traffic mapped into a class with no action and all other traffic being inspected in the class-default:

class-map type inspect match-all web
match protocol http
!
policy-map type inspect InToOut
class type inspect web
class class-default
inspect
!
zone security Inside
zone security Outside
zone-pair security InToOut source Inside destination Outside
service-policy type inspect InToOut
As expected, HTTP traffic (port 80) was the only traffic blocked by the firewall.

Use access-lists to filter IP packets with IP options

In the security advisory detailing the Crafted IP Option Vulnerability (a nasty bug that allows an intruder to reset your router with a ping packet), the authors forgot to mention a great tool available in IOS release 12.3(4)T (and integrated in 12.4): the ability to filter packets with IP options in an access-list.I've used this proof-of-concept access list that logs all ping packets with IP options:

ip access-list extended LogOptions
permit icmp any any echo option any-options log
permit icmp any any echo-reply option any-options log
permit ip any any
After an extended ping from my Windows XP workstation, the router (as expected) logged the offending packets:
%SEC-6-IPACCESSLOGDP: list LogOptions permitted icmp 10.0.0.10 -> 172.16.0.12 (8/0), 1 packet
%SEC-6-IPACCESSLOGDP: list LogOptions permitted icmp 10.0.0.10 -> 172.16.0.12 (8/0), 3 packets
Note: you can easily generate ICMP packets with IP options with the -r, -s, -j or -k options of the Windows XP ping program (and I am positive Linux ping has equivalent options).

Reading the Cisco advisory, the IP access-list that would drop all dangerous packets would be similar to this one (not tested, use at your own discretion):
ip access-list extended ProtectRouter
deny icmp any any echo option any-options
deny icmp any any information-request option any-options
deny icmp any any timestamp-request option any-options
deny icmp any any mask-request option any-options
deny pim any any option any-options
deny 113 any any option any-options
deny tcp any any eq 465 option any-options
permit ip any any

Multihomed small sites

My February IP corner article, Small Site Multihoming, describes how you can reliably connect to two Internet Service Providers without running BGP or owning part of public IP address space. You can also view recording of an online session (registration required) I did explaining the details.

The solution in the article relies on two IOS technologies: multiple Network Address Translation (NAT) pools and reliable static routes (static routes tied to SLA measurements).

Don't use copy commands in EEM applets

I've tried to be a "good citizen" in one of my Embedded Eveng Manager (EEM) applications and used the action 1.1 cli command "copy running-config startup-config" instead of the (supposedly obsolete) action 1.1 cli command "write memory" command. However, as the copy command generates console prompts (unless you turn them off with the file prompt quiet command), it immediately hung my EEM applet.

Moral of the story: use write memory :)