One of the hardest troubleshooting problems within an MPLS VPN network has always been finding a broken LSP. While you could (in theory) use the IP ping or traceroute (assuming all hops support ICMP extensions for MPLS), the results are not always reliable… and interpreting them is not so easy. For example, after I've disabled LDP on an interface with the no mpls ip configuration command, the routers in the LSP path still reported outgoing MPLS labels in ICMP replies for a few seconds (until the LDP holddown timer expired on both ends of the link).

As a side note, would you deduce from the printout that the break in the LSP path happened on the router with the IP address 192.168.201.1?

Due to the way Cisco routers behave when dropping packets with an inbound access list, whenever you use access lists to protect the router from the outside attacks (or port scans), the protected ports (even though they're not active on the router) will appear filtered (some scanners might use the term stealth), which is almost an invitation to a determined hacker.

Sometimes (it depends on the application you're protecting) you can configure application-layer protection in Cisco IOS. For example, you can protect HTTP server with ip http access-class global configuration command or the Telnet server with the access-class in line configuration command (and BGP will not accept incoming TCP SYN packets unless you've configured a BGP neighbor). The access-class configuration causes the incoming request to be rejected within application (in control plane after the TCP stack), resulting in TCP RST packet being sent back. The port scanner thus reports the protected TCP port as closed.

Under some circumstances, you might want to tune the ARP timers on the router (for example, when using ARP as a keepalive mechanism to detect whether the host is up). Unfortunately, although you can set the per-interface arp timeout in seconds, the actual timer resolution is in minutes. For example, if you set the ARP timeout to 10 seconds, the router will age the ARP entries once per minute.

When I was testing the inspection of router-generated traffic, I wanted to block and log all incoming traffic (apart from inspect-generated conduits, obviously) with a simple access-list:

access-list 102 deny ip any any log

Unfortunately, the port numbers in the logging printout were always zero:

%SEC-6-IPACCESSLOGP: list 102 denied udp 10.0.0.1(0) -> 192.168.1.3(0), 1 packet

The reason for this behavior is very simple: unless a line in the IP ACL matches on the layer-4 port numbers, the router does not inspect them; the log action thus has no port number to show in the syslog printout.

To fix the printout, you have to force the router to inspect the layer-4 port numbers. If you still want to block-and-log all traffic, the minimum access-list achieving this goal is the following:

access-list 102 deny   udp any gt 0 any gt 0 log
access-list 102 deny   tcp any gt 0 any gt 0 log
access-list 102 deny   ip any any

In a previous post I've been writing about the inability to clean the ARP cache due to cached CEF adjacencies. As it turns out, this behavior has another side effect: the router will automatically refresh all ARP entries (and CEF adjacencies) as they expire from the ARP cache. This might become a problem on high-end devices with a lot of directly connected hosts if you set the arp timeout to a low value.

Recently I was trying to figure out what the various port states reported by Nmap really mean. This is what's actually going on:

• If a packet is intercepted by a router's access-list, the router sends back an ICMP administratively prohibited packet. This is reported as filtered port by Nmap (and probably as stealth port by some other scanners).
• If you do a TCP SYN scan of a router and the scanned port is not active, the router sends back TCP RST packet. This is reported as closed port.
• If you perform a UDP scan of a router, the router sends back ICMP port unreachable message if the UDP application is not active. This is reported by Nmap as filtered port (even though in most cases it should be equivalent to closed TCP port).
• In some cases, the router simply doesn't reply to UDP scans (for example, if you scan the discard service). This is reported as Open¦Filtered (as the scanner cannot reliably determine whether the probe was dropped due to a filter or simply not replied to).

Note: In any case, UDP scans are way more unreliable than TCP scans due to connectionless nature of UDP.

It's a bit embarrassing: I've posted a hint on how to execute commands after a router reload and Aamer figured out something I've wanted to have for years: the ability to automatically start debugging after the router reload.

The easiest way to remove all settings from an interface is to use the default interface configuration command. For example, if you've configured Frame Relay interface with subinterfaces ...

interface Serial0/0/0
 no ip address
 encapsulation frame-relay
 load-interval 60
!
interface Serial0/0/0.100 point-to-point
 bandwidth 2000
 ip address 172.16.1.1 255.255.255.252
 ip load-sharing per-packet
 ip ospf cost 50
 frame-relay interface-dlci 100

... and have erase all interface-specific configuration, the ...

rtr(config)#default interface serial 0/0/0
Building configuration...

Interface Serial0/0/0 set to default configuration

... gets you there. As you can see, after the configuration change, the main interface has no IP address and the subinterface is deleted.

a1#show ip interfaces brief
Interface          IP-Address  OK? Method Status     Protocol

... non-relevant lines deleted ...

Serial0/0/0        unassigned  YES TFTP   up         up
Serial0/0/0.100    unassigned  YES manual deleted    down

A while ago a reader has asked me whether you could modify an IP access-list when the interface IP address changes. While that's definitely doable with Tcl and Embedded Event Manager, it's not a trivial task, so I've tried to understand why he would need such a functionality.

The answer was quite interesting: he's running NTP on his firewall router and thus needs to accept incoming NTP responses from an external NTP server. While that could be easily achieved with the following configuration (only the relevant bits-and-pieces are shown), he didn't want to make the access-list too generic (allowing NTP from the external server to any IP address).

ip inspect name DEFAULT100 tcp
ip inspect name DEFAULT100 udp
!
interface Dialer0
 description $FW_OUTSIDE$
 ip access-group 102 in
 ip inspect DEFAULT100 out
!
access-list 102 remark #### Dialer0 incoming ####
access-list 102 remark #### non-relevant lines deleted
access-list 102 permit udp host 1.2.3.4 eq ntp any eq ntp

This problem nicely illustrates a broader issues: the router does not inspect it's own traffic and thus does not prepare conduits for the return packets; you have to specify all the return traffic you're expecting in the incoming access list. This drawback has been fixed in IOS release 12.3(14)T with the introduction of the Inspection of Router-Generated Traffic feature. In our scenario you only need to change the inspect rules:

ip inspect name DEFAULT100 tcp router-traffic
ip inspect name DEFAULT100 udp router-traffic

... and the router synchronizes to an external NTP server:

sp#show ip inspect sessions
Established Sessions
 Session 474032B4 (192.168.1.3:123)=>(10.0.0.1:123) udp SIS_OPEN
sp#
01:04:34: %NTP-5-PEERSYNC: NTP synced to peer 10.0.0.1
01:04:34: %NTP-6-PEERREACH: Peer 10.0.0.1 is reachable

Note: This article is part of You've asked for it series.

It makes perfect sense in hindsight, but I was nonetheless pleasantly surprised: when the router acting as a DHCP client (configured with the ip address dhcp interface configuration command) receives the DHCP reply packet containing the default gateway option (option #3), it installs a static default route toward that next-hop.

Even better, the default route is installed with the administrative distance 254 (floating static route), making sure that the default route you’ve configured manually or the default route received via a routing protocol are not overwritten.

The IOS documentation claims that the quiet mode the router enters after a series of login failures blocks all telnet (or ssh) sessions as well as HTTP requests. Unfortunately the latter is wrong; you can execute any HTTP request on the router during the quiet mode.

If you want to block HTTP requests during the quiet mode, you can use EEM applets to change the HTTP server configuration when the quiet mode is started and completed.
First you need to configure a standard numbered IP access list that will be used to block HTTP requests during the quiet mode (the ip http access-class command accepts only numbered ACLs), for example:

access-list 95 deny any log

Then you define two EEM applets: one that triggers when the router enters the quiet mode (matching the SEC_LOGIN-1-QUIET_MODE_ON syslog message) and another that runs when the quiet mode is finished (triggered with the SEC_LOGIN-5-QUIET_MODE_OFF). Both applets modify the router configuration, changing the access-list used in ip http access-class configuration command.

event manager applet EnterQuietMode
 event syslog occurs 1 pattern "SEC_LOGIN-1-QUIET_MODE_ON" period 1
 action 1.0 cli command "configure terminal" 
 action 1.1 cli command "ip http access-class 95" 
 action 2.0 syslog msg "Entered Quiet mode on HTTP server"
!
event manager applet ExitQuietMode
 event syslog occurs 1 pattern "SEC_LOGIN-5-QUIET_MODE_OFF" period 1
 action 1.0 cli command "configure terminal"
 action 1.1 cli command "ip http access-class 70"
 action 2.0 syslog msg "Exiting Quiet mode on HTTP server"

If you use UDP-based syslog servers, you might have noticed that they miss a message or two generated during a router reload (particularly when the syslog server is on a directly connected LAN). The reason is simple - when the first message is sent to the syslog server, its MAC address is not yet in the router's ARP cache and the message is dropped. To prevent the message loss, you can use the logging server-arp configuration command (introduced in IOS release 12.3T), making sure that the router sends ARP request to the configured syslog server(s) before generating the first syslog messages.

Note: this issue does not occur when using TCP as the syslog transport mechanism.

With the introduction of DNS views in IOS release 12.4(9)T, a number of additional DNS-related configuration commands were introduced. As IOS still supports all the older configuration commands (and the DHCP-acquired DNS servers), the results are not as obvious as one would hope. The IOS documentation is pretty explicit (a nice surprise :), but I still had a bit of a headache figuring it all out, so you might be in the same position.

Let's start with the DNS resolvers (the DNS name servers the router itself uses when it needs to change a name into an IP address or vice versa):

• You can configure the DNS resolvers with the ip name-server global configuration command or the domain name-server configuration command within the ip dns view default.
• Both lists are merged and combined with the IP addresses acquired from the DHCP reply messages to get the final list of the DNS resolvers. You can inspect the final list with the show ip dns view default command.
• The DNS name servers from the DHCP replies never appear in the router configuration.
• If the same IP address is specified with the ip name-server and domain name-server command, only the global (ip name-server) command will appear in the router configuration.

The rules for DNS forwarders (the DNS servers the router uses to answer incoming DNS queries) are a bit different:

• The DNS view the current DNS query should use is determined based on view-list assigned to incoming interface (with the ip dns view-group interface configuration command) or the global view-list (specified with the ip dns server view-group global configuration command).
• The incoming query is (sequentially) sent to the IP addresses configured as dns forwarders in the selected view. No other DNS servers are used.
• If the selected view has no dns forwarders, but contains domain name-servers, they are used as forwarders.
• If the selected view has no dns forwarders or domain name-servers, the query is forwarded as IP broadcast ... unless the selected view is the default view, in which case the first DNS server computed according to the the previous bullet list is used (only one DNS server is used in this case, even if you have configured multiple DNS servers with the ip name-server configuration command).

After you've configured the Warm Reload, you can also perform warm IOS upgrade/downgrade (assuming that you already run at least the IOS release 12.3(11)T or 12.4). The Warm Upgrade functionality loads the new IOS image into the main memory, decompresses it and starts it, significantly reducing the downtime (in my case, a 2800 router reloaded in 62 seconds as compared to 415 seconds it took to load the image from a locally-attached server).

Apart from the downtime reduction, the warm upgrade (requested with the reload warm file url command) has a number of other benefits:

• The new image does not have to be stored in flash
• You don't have to change the boot image with the boot system command
• If the new image crashes, the router will revert to the original IOS image stored in flash

Testing Embedded Event Manager (EEM) Tcl policies is a convoluted process:

• Source file is usually edited on a general-purpose workstation.
• The file has to be downloaded to router's local storage (EEM does not register non-local policies).
• The new version of the EEM policy has to be registered with EEM with event manager policy configuration command
• After all these steps, the new policy can be tested.

While you can use EEM applet to automate this process, slightly more flexible approach (you can specify the policy name to be replaced) can be implemented with Tcl script:

set policy [lindex $argv 0]
set source "tftp://10.0.0.10/tcl/" # replace with your host and directory
set destination "nvram:" # replace with local storage device
if {[string equal $policy ""]} {
return -code error "expected policy name"
}
puts "replacing policy: $policy"
ios_config "file prompt quiet"
ios_config "no event manager policy $policy" ""
exec "copy $source$policy $destination$policy"
ios_config "event manager policy $policy"
ios_config "no file prompt quiet"

To use the script, follow these steps:

• Save the script in a .tcl file (for example, changePolicy.tcl)
• Change the script parameters (remote host and local storage)
• Save the .tcl file to your router's local storage (you can also run it from a remote server)
• Configure a command alias, for example alias exec eem tclsh flash:changePolicy.tcl testPolicy.tcl

Now you can replace the target EEM Tcl policy with a simple eem command.

Alternatively, if you define alias exec eem tclsh flash:changePolicy.tcl, you can specify policy name as an argument to the eem command, for example eem testPolicy.tcl.