Harold Arley Morales has asked an interesting question:

What's the difference between Cisco Express Forwarding and Cisco MLS? Is Cisco's implementation of MLS standardized?

CEF is a routing table lookup mechanism. Instead of doing a lookup in the main IP routing table (displayed with the show ip route), the router does a lookup in a fully computed non-recursive version of the IP routing table (Forwarding Information Base - FIB) with layer-2 next-hop information attached to it (adjacency table).

MLS is a caching mechanism (similar to Netflow) that offloads layer-3 processing from the routing component into layer-2 ASICs that cannot perform full-blown layer-3 switching. When the layer-2 engine detects a single IP packet traversing multiple VLANs, the MLS populates the cache with the flow details and the subsequent packets belonging to the same flow (same source/destination IP addresses and port numbers ...) are switched without going through all the layer-3 mechanisms (for example, access lists). The Multilayer Switching Overview document gives you additional details.

The MLS uses a proprietary protocol (MLSP) through which the layer-2 switches identify routers.

This article is part of You've asked for it series.

Update 2008-12-08: Ofer Granit sent me the following information: according to Troubleshooting IP Multilayer Switching document, Supervisor Engine 2 and Supervisor Engine 720 no longer use MLS but rely exclusively on CEF to perform layer-3 forwarding.

When testing the ERM functionality that together with an EEM applet generates SNMP traps whenever the CPU load exceeds predefined thresholds, I started to wonder what the snmp-server enable traps cpu threshold command does.

After lenghty conversation with uncle Google and Cisco documentation, I found that there's another way to detect and report high CPU load in Cisco IOS: the CPU threshold notification introduced in IOS release 12.3T.

I've just started to read the AXP (Linux on router module) documentation and found the set of supported programming languages somewhat intriguing. I don't like Tcl at all (and I am glad someone within Cisco finally acknowledged the existence of PERL), but if they made us jump through the hoops to get scripting capabilities on IOS, they should include it on AXP as well.

Gernot Nusshall has asked an interesting question:

How could I configure the EEM to send an SNMP trap when the cpu load (interval=30sec) is higher than 30%?

My first solution was to enable resource policy traps with the snmp-server enable traps resource-policy, but this feature was introduced in 12.4(15)T and I am not sure everyone is willing to run the latest-and-greatest IOS code. Furthermore, it looks like the traps are sent only for resource policies defined through the ERM MIB; I was not able to generate a trap from a manually configured resource policy. Obviously it was time for another EEM applet.

Ten days ago, the industry press was buzzing with the news of the IOS rootkit developed by Sebastian Muniz. At that time I wrote “Personally I doubt it would go beyond Tcl scripts that we already know about” … and now it's time to admit that:

1. I was wrong.
2. I'm really impressed.

Although the rootkit was just a proof of concept (which is usually enough for a white-hat researcher), it does demonstrate that you can (with proper skills, tools and lots of patience) reverse-engineer IOS, write your own code and insert it into IOS image.

The rootkit presentation prompted Cisco to generate an excellent document describing how to detect patched IOS images and the precautions you can take to ensure an intruder does not get access to your devices.

On the other hand, I was bitterly disappointed by the lack of coverage from the "industry press". There was speculation that Cisco released three patches in anticipation of the presentation (anyone who looked into what those patches were would easily find out that two of them were not IOS related) and a few notable exceptions correctly describing the situation, but some publications that were very loud before the presentation forgot to tell their readers that the threat was "slightly" over-rated. Of course, the lack of interest in non-sensational news has already started conspiracy theories.

If you want to have more details, read a down-to-earth description of the presented rootkit by Nicolas Fischbach.

Xavier has mentioned an interesting undocumented command in his comment to the “Continuous display of top CPU processes” post: after you execute terminal exec prompt timestamp, every show command displays current time and CPU utilization before the requested printout.

Sebastian Majewski has found an interesting feature: if you use the network route-map BGP configuration command to originate BGP prefixes and use the match conditions within the route-map, BGP inserts the IP prefix in the BGP table only if the source route in the IP routing table satisfies the route-map conditions.

When you have to monitor which processes consume router’s CPU over a period of time, a Tcl script that emulates the Unix top command might come handy. The following Tcl script continuously displays top 20 Cisco IOS processes and refreshes the update every 5 seconds.

Installation

• Download the source file into flash:top.tcl.
• Configure alias exec top tclsh flash:top.tcl.
• Invoke with top.

Usage guidelines

Usage: top [ 5sec | 1min | 5min ]

The script changes the escape character to Ctrl/C. Use terminal escape default to restore default settings

Source code

#
# title:    Emulate the Unix top command
# name:     top.tcl
# desc:     The script displays top CPU processes every 5 seconds
#
# ios config:
#
#           * download the file into flash:top.tcl
#           * configure alias exec top tclsh flash:top.tcl
#
#           invoke with top [5sec|1min|5min]
#

set IOS [string equal $tcl_platform(os) "Cisco IOS"];

if { $IOS } { 
  exec "terminal international"; 
  exec "terminal escape 3";
}

set arg [lindex $argv 0];
if { [string length $arg] == 0 } { set arg "5sec" } ;
if { [lsearch -exact { 5sec 1min 5min } $arg] < 0 } {
  puts {Usage: top [5sec|1min|5min]};
  return 0;
}

fconfigure stdout -buffering none;

while {1} {
  set lines [split [exec "show process cpu sorted $arg | exclude 0.00% +0.00% +0.00%"] "\n"];

  puts -nonewline "\033\[2J\033\[H";
  for { set lc 1 } { $lc < 23 } { incr lc } {
    set curline [lindex $lines $lc];
    if { [string length $curline] > 0 } { puts "$curline"; }
  }
  puts -nonewline "\nBreak with Ctrl/C --> ";
  after 5000;
}

Another interesting question I've received:

Can you please recommend me literature explaining the architecture of Cisco routers and switches (buffers, control-plane, forwarding-plane, process switching …)

The best one I've found so far is the Inside Cisco IOS Software Architecture, but it's a bit old, so if you've found something better, please comment.

Full disclosure: if you click on the link above and buy the book, I might eventually get $1.76 from Amazon.

The undesired intermittent bridging behavior of Cisco 871 using old ROMMON software can lead to hard-to-diagnose problems if you're connected to an Internet access network through a cable modem that accepts only a single MAC address. The right sequence of events can leave the router/modem combination in a state with no external connectivity requiring a modem power-cycle:

1. The router and the cable modem are power-cycled.
2. The router starts to bridge between all LAN interfaces, effectively connecting inside workstations directly to the cable modem.
3. One of the workstations could detect a LAN failure (due to router reload) and restart the DHCP process (a Windows XP host would definitely do that).
4. The DHCP requests from the workstation are bridged straight to the cable modem which caches the workstation's MAC address and forwards the DHCP request.
5. The workstation is assigned a public IP address (at this time, the workstation is connected directly to Internet and thus vulnerable).
6. The router loads Cisco IOS and reinitializes the Ethernet interfaces. Bridging between internal and external interfaces is stopped.
7. The router sends DHCP request on the outside interface, but the modem ignores it, as the MAC address of the DHCP request differs from the previously cached one.

In most cases, the cable modem has to be power-cycled to lose the cached MAC address.

This behavior can be observed only if the router and the cable modem are reset at the same time and the cable provider doesn't care much about MAC security and allows the modem to learn the MAC address. If you reset only the cable modem, the router is not bridging (no problem); if you reset just the router, the cable modem still caches the router's MAC address and ignores the DHCP request from the inside workstation(s).

Multihoming to a single ISP is a design scenario in which a customer uses multiple Internet connections to the same Internet Service Provider. This design provides resilience against link and device failures, but does not provide protection against major outages within the Service Provider network.

There are three major decisions to be made when designing multihoming to single ISP:

• Will the customer use provider-assigned or provider-independent address space?
• Should the customer use static or dynamic routing with the ISP?
• When using dynamic routing with BGP, does the customer need its own public autonomous system?

In the last days, industry journalists have started to make a big fuzz about a Cisco IOS rootkit that someone is going to present in a few days. Personally I doubt it would go beyond Tcl scripts that we already know about (OK, maybe it's EEM-based so it doesn't need a VTY and maybe it starts at router reload) … but I might be really surprised.

However, the Cisco's response to this announcement (which was basically saying "we haven't seen anything new yet") included a nice gem: a link to the Cisco Guide to Harden Cisco IOS Devices document.

Keep in mind: Use private IP addresses, AS numbers and domain names in all technical documentation you're producing (unless, of course, you're describing an actual network). If you're forced to use public addresses or AS numbers (for example, to illustrate how the neighbor remote-private-as command works), you should clearly state that they are imaginary.

You can safely use:

The inability of Control Plane host interface to detect inbound OSPF packets has prompted Sebastian and myself to search for more documentation and conduct further tests. Sebastian already had a working configuration from which he could infer most of the configuration rules and he also found the well-written Understanding CPPr document on CCO. Together with the tests I ran in my router lab, we're pretty confident the CPPr inbound packet classification rules are (approximately) as follows:

Use the latest 12.4T software (at least 12.4(15)T5) if you want reliable CPPr operation.

• control-plane aggregate service-policy disables any control-plane subinterface service policies.
• If you want to use the per-subinterface (host, transit and cef-exception) policies, you have to remove the inbound service policy from the control-plane aggregate path.
• Routed packets that cannot be CEF-switched (have to be punted to another switching mechanism) are classified as transit packets.
• Local multicast packets with destination IP addresses within IP prefix 224.0.0.0/24 and packets with TTL <= 1 are classified as transit packets in 12.4(15)T5. These packets will be classified as cef-exception packets in the future (see the Understanding CPPr document).
• Unicast packets without IP options addressed to the router and having TTL > 1 are classified as host packets.
• Non-IP traffic (ARP, Frame Relay keepalives, CDP ...) is classified as cef-exception.

The TTL-related rules explain why the router classifies IBGP packets as host packets and EBGP packets as transit packets. As soon as you configure neighbor ebgp-multihop on the router router, inbound EBGP packets become host packets.

Years ago EIGRP and OSPF had strong presence in large enterprise networks, BGP was used solely by Internet Service providers and IS-IS was a rarity (and there were people using Banyan Vines).

The situation has probably changed over the last years, I would (sadly) expect EIGRP to decline and (happily) BGP to grow. Let's figure it out; please respond to this week's readers' poll. Of course you can choose more than one routing protocol.