Building network automation solutions

9 module online course

Start now!

Blog Posts in May 2008

Much ado about rootkits

Ten days ago, the industry press was buzzing with the news of the IOS rootkit developed by Sebastian Muniz. At that time I wrote “Personally I doubt it would go beyond Tcl scripts that we already know about” … and now it's time to admit that:
  1. I was wrong.
  2. I'm really impressed.
Although the rootkit was just a proof of concept (which is usually enough for a white-hat researcher), it does demonstrate that you can (with proper skills, tools and lots of patience) reverse-engineer IOS, write your own code and insert it into IOS image.

The rootkit presentation prompted Cisco to generate an excellent document describing how to detect patched IOS images and the precautions you can take to ensure an intruder does not get access to your devices.

On the other hand, I was bitterly disappointed by the lack of coverage from the "industry press". There was speculation that Cisco released three patches in anticipation of the presentation (anyone who looked into what those patches were would easily find out that two of them were not IOS related) and a few notable exceptions correctly describing the situation, but some publications that were very loud before the presentation forgot to tell their readers that the threat was "slightly" over-rated. Of course, the lack of interest in non-sensational news has already started conspiracy theories.

If you want to have more details, read a down-to-earth description of the presented rootkit by Nicolas Fischbach.

see 8 comments

Conditional BGP Route Origination

Sebastian Majewski has found an interesting feature: if you use the network route-map BGP configuration command to originate BGP prefixes and use the match conditions within the route-map, BGP inserts the IP prefix in the BGP table only if the source route in the IP routing table satisfies the route-map conditions.

The following text written by Ivan Pepelnjak in 2008 was originally published on CT3 wiki. That web site became unreachable in early 2019. We retrieved the original text from the Internet Archive, cleaned it up, updated it with recent information if necessary, and republished it on blog on December 28, 2020
read more add comment

Continuous display of top CPU processes

When you have to monitor which processes consume router’s CPU over a period of time, a Tcl script that emulates the Unix top command might come handy. The following Tcl script continuously displays top 20 Cisco IOS processes and refreshes the update every 5 seconds.

The following text written by Ivan Pepelnjak in 2008 was originally published on CT3 wiki. That web site became unreachable in early 2019. We retrieved the original text from the Internet Archive, cleaned it up, updated it with recent information if necessary, and republished it on blog on November 17, 2020


  • Download the source file into flash:top.tcl.
  • Configure alias exec top tclsh flash:top.tcl.
  • Invoke with top.

Usage guidelines

Usage: top [ 5sec | 1min | 5min ]

The script changes the escape character to Ctrl/C. Use terminal escape default to restore default settings

If anyone discovered a reliable technique that detects a keypress event (= character available on stdin) in the Tcl loop, please let me know. The Ctrl/C solution is a kludge.

Source code

# title:    Emulate the Unix top command
# name:     top.tcl
# desc:     The script displays top CPU processes every 5 seconds
# ios config:
#           * download the file into flash:top.tcl
#           * configure alias exec top tclsh flash:top.tcl
#           invoke with top [5sec|1min|5min]

set IOS [string equal $tcl_platform(os) "Cisco IOS"];

if { $IOS } { 
  exec "terminal international"; 
  exec "terminal escape 3";

set arg [lindex $argv 0];
if { [string length $arg] == 0 } { set arg "5sec" } ;
if { [lsearch -exact { 5sec 1min 5min } $arg] < 0 } {
  puts {Usage: top [5sec|1min|5min]};
  return 0;

fconfigure stdout -buffering none;

while {1} {
  set lines [split [exec "show process cpu sorted $arg | exclude 0.00% +0.00% +0.00%"] "\n"];

  puts -nonewline "\033\[2J\033\[H";
  for { set lc 1 } { $lc < 23 } { incr lc } {
    set curline [lindex $lines $lc];
    if { [string length $curline] > 0 } { puts "$curline"; }
  puts -nonewline "\nBreak with Ctrl/C --> ";
  after 5000;
see 10 comments

Router architecture books

Another interesting question I've received:
Can you please recommend me literature explaining the architecture of Cisco routers and switches (buffers, control-plane, forwarding-plane, process switching …)
The best one I've found so far is the Inside Cisco IOS Software Architecture, but it's a bit old, so if you've found something better, please comment.

Full disclosure: if you click on the link above and buy the book, I might eventually get $1.76 from Amazon.

see 5 comments

Cable modem problems with Cisco 871

The undesired intermittent bridging behavior of Cisco 871 using old ROMMON software can lead to hard-to-diagnose problems if you're connected to an Internet access network through a cable modem that accepts only a single MAC address. The right sequence of events can leave the router/modem combination in a state with no external connectivity requiring a modem power-cycle:
  1. The router and the cable modem are power-cycled.
  2. The router starts to bridge between all LAN interfaces, effectively connecting inside workstations directly to the cable modem.
  3. One of the workstations could detect a LAN failure (due to router reload) and restart the DHCP process (a Windows XP host would definitely do that).
  4. The DHCP requests from the workstation are bridged straight to the cable modem which caches the workstation's MAC address and forwards the DHCP request.
  5. The workstation is assigned a public IP address (at this time, the workstation is connected directly to Internet and thus vulnerable).
  6. The router loads Cisco IOS and reinitializes the Ethernet interfaces. Bridging between internal and external interfaces is stopped.
  7. The router sends DHCP request on the outside interface, but the modem ignores it, as the MAC address of the DHCP request differs from the previously cached one.

In most cases, the cable modem has to be power-cycled to lose the cached MAC address.

This behavior can be observed only if the router and the cable modem are reset at the same time and the cable provider doesn't care much about MAC security and allows the modem to learn the MAC address. If you reset only the cable modem, the router is not bridging (no problem); if you reset just the router, the cable modem still caches the router's MAC address and ignores the DHCP request from the inside workstation(s).

add comment

Multihoming to a Single ISP

Multihoming to a single ISP is a design scenario in which a customer uses multiple Internet connections to the same Internet Service Provider. This design provides resilience against link and device failures, but does not provide protection against major outages within the Service Provider network.

There are three major decisions to be made when designing multihoming to single ISP:

  • Will the customer use provider-assigned or provider-independent address space?
  • Should the customer use static or dynamic routing with the ISP?
  • When using dynamic routing with BGP, does the customer need its own public autonomous system?
read more add comment

Guide to Harden Cisco IOS Devices

In the last days, industry journalists have started to make a big fuzz about a Cisco IOS rootkit that someone is going to present in a few days. Personally I doubt it would go beyond Tcl scripts that we already know about (OK, maybe it's EEM-based so it doesn't need a VTY and maybe it starts at router reload) … but I might be really surprised.

However, the Cisco's response to this announcement (which was basically saying "we haven't seen anything new yet") included a nice gem: a link to the Cisco Guide to Harden Cisco IOS Devices document.
see 10 comments

Using IP Prefixes, AS Numbers and Domain Names in Examples

Keep in mind: Use private IP addresses, AS numbers and domain names in all technical documentation you're producing (unless, of course, you're describing an actual network). If you're forced to use public addresses or AS numbers (for example, to illustrate how the neighbor remote-private-as command works), you should clearly state that they are imaginary.

You can safely use:

read more see 5 comments

Control Plane Protection inbound packet classification

The inability of Control Plane host interface to detect inbound OSPF packets (and the flurry of comments that followed my blog post) has prompted Sebastian and myself to search for more documentation and conduct further tests. Sebastian already had a working configuration from which he could infer most of the configuration rules and he also found the well-written Understanding CPPr document on CCO. Together with the tests I ran in my router lab, we're pretty confident the CPPr inbound packet classification rules are (approximately) as follows:

Use the latest 12.4T software (at least 12.4(15)T5) if you want reliable CPPr operation.

  • control-plane aggregate service-policy disables any control-plane subinterface service policies.
  • If you want to use the per-subinterface (host, transit and cef-exception) policies, you have to remove the inbound service policy from the control-plane aggregate path.
  • Routed packets that cannot be CEF-switched (have to be punted to another switching mechanism) are classified as transit packets.
  • Local multicast packets with destination IP addresses within IP prefix and packets with TTL <= 1 are classified as transit packets in 12.4(15)T5. These packets will be classified as cef-exception packets in the future (see the Understanding CPPr document).
  • Unicast packets without IP options addressed to the router and having TTL > 1 are classified as host packets.
  • Non-IP traffic (ARP, Frame Relay keepalives, CDP ...) is classified as cef-exception.

The TTL-related rules explain why the router classifies IBGP packets as host packets and EBGP packets as transit packets. As soon as you configure neighbor ebgp-multihop on the router router, inbound EBGP packets become host packets.

see 4 comments

Which routing protocol do you use?

Years ago EIGRP and OSPF had strong presence in large enterprise networks, BGP was used solely by Internet Service providers and IS-IS was a rarity (and there were people using Banyan Vines).

The situation has probably changed over the last years, I would (sadly) expect EIGRP to decline and (happily) BGP to grow. Let's figure it out; please respond to this week's readers' poll. Of course you can choose more than one routing protocol.
see 11 comments

Cisco 851 and 871 bridge between LAN and WAN interfaces during boot process

Euphrates Greene sent me a report of a very annoying “functionality” of Cisco 851/871: they're bridging between the inside (LAN) ethernet and outside (WAN) ethernet interfaces while they're running the ROMMON code (from the reload/power-up throughout the software decompression process until the control is transferred to the Cisco IOS). It's worth mentioning that these routers are commonly used as SOHO firewalls and that the internal LAN is exposed while the router is in the bridging mode.

Our security experts have replicated the behavior and reported it to Cisco PSIRT. Fortunately it's a known vulnerability, documented as CSCsd60259 (release note is available on CCO to registered users) and fixed with a ROMMON upgrade.

New routers are shipped with new ROMMON version, so you shouldn't be seeing this behavior on brand new boxes … but one cannot help but wonder why such a nasty behavior was not documented as a field notice/security advisory.
add comment

RTBH links (and thanks for the acronym :)

One of the comments to my Sunday post mentioned RTBH. Obviously I'm not geeky enough, as I had to ask uncle Google for help (but don't worry, I'll work on my geekiness factor :).

The search results produced a few very interesting links, among them a well-structured presentation on RTBH that refers to a paper describing how you can detect remote DoS attacks with the backscatter analysis (assuming the attackers are randomly spoofing source IP addresses).
see 3 comments

How do you know you're an SP-geek

  1. You're creating a multi-AS BGP test lab on Sunday evening;
  2. The core AS is running 12.2SRC code;
  3. You insert a P-router in the core network ... because every large network has P-routers;
  4. You create BGP session templates instead of configuring two parameters of a few IBGP neighbors;
  5. You configure MPLS in the core network instead of using BGP on all routers ... because it saves you a few BGP sessions ... and that's the way things should be done anyway;
  6. When configuring OSPF, you define inter-AS links as passive interfaces ... not because you're running OSPF in the other AS but for security reasons :)
  7. ... add your comment here ...
see 6 comments

Please comment: Is asymmetric routing harmful?

We've always been trying to minimize asymmetric routing, in both design and implementation phase, as it impacts a number of IP services/features, including:

  • Network Address Translation;
  • Content-based Access Control (CBAC);
  • Reflexive access lists;
  • Redundant firewalls (at least until recently);
  • IP Multicast;

In some scenarios, asymmetric routing can impact delay/jitter and consequently the perceived quality of service.

However, asymmetric routing is a reality within the Internet (it's close to impossible to guarantee symmetric routing even for multi-homed end users) and it might even help in some scenarios (low-speed/low-delay upstream link with high-speed/high-delay downstream link).

What's your opinion? Is asymmetric routing harmful? Should we strive to avoid it ... or do you just accept it as one of facts of life?

see 10 comments

OSPF bypasses Control Plane Host Subinterface

I wanted to implement a mechanism that would automatically (using EEM) block unstable OSPF neighbors. Once you identify the neighbors to block (this should be the hard part), blocking them is easy if you're running point-to-point interfaces (you just make the interface passive), but blocking a single neighbor on a multi-access interface is a royal pain. I didn't want to use the access lists, as it would be very hard to integrate OSPF-specific filters with existing incoming access-lists configured on the interfaces. Control Plane Protection looked like the ideal tool to use; if I could drop certain inbound IP packets (OSPF hello packets) based on their source IP address (= unstable neighbor) and IP protocol (= OSPF), they would never get to the OSPF process and the adjacency would not form, resulting in a more stable network.

Update: After the comment from William Chu, I've tested 12.4 mainstream release. OSPF is blocked as configured. Next I've re-read the documentation … and found that one of the documented restrictions is that the host subinterface only filters UDP and TCP traffic. Configuring the service policy on the aggregate path (the control-plane keyword with no options) worked.

Before trying to figure out the integration between the SYSLOG messages and router configuration changes, I performed an easy test: I tried blocking all OSPF traffic in the host control-plane (the one controlling packets received by the IOS processes) with the following configuration:

class-map match-all BlockOSPF
 match access-group name BlockOSPF
policy-map ControlPlane
 class BlockOSPF
ip access-list extended BlockOSPF
 permit ospf any any
control-plane host
 service-policy input ControlPlane
However, according to the show commands, the service policy did not identify any packets as belonging to the BlockOSPF class and the OSPF adjacencies were not affected:
C1#show policy-map control-plane host
 Control Plane Host

  Service-policy input: ControlPlane

    Class-map: BlockOSPF (match-all)
      0 packets, 0 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: access-group name BlockOSPF

    Class-map: class-default (match-any)
      5 packets, 400 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: any
C1#show ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface 1 FULL/DR 00:00:30 FastEthernet0/0 0 FULL/ - 00:00:33 Serial1/0.101 0 FULL/ - 00:00:33 Serial1/0.100
After a few more tests, I had to conclude that the Control Plane Protection using host subinterface does not work on OSPF packets (and it might does not work on other non-TCP/UDP traffic either). Consequently you cannot protect your router from a DoS attack coming through an interface on which you have to run OSPFTo filter non-TCP/UDP traffic, use the aggregate path control plane protection.
see 16 comments

The “fallback global” VRF option does not exist in Cisco IOS

Cheng sent me an interesting question:
I'm reading your book MPLS and VPN Architectures and I've found the ip vrf forwarding name fallback global command in the “Additional Lookup in the Global Routing Table” section. I can only find this command in Junos, but not in IOS.

… and he was right. When we were writing the book, we described several features that were still in development as it looked like they would be in the production code by the time the book was published. Many of them made it into the public IOS releases (for example, the Carrier's Carrier architecture), but some of them (like this command) simply vanished from the surface.

However, it looks like the engineers that switched from Cisco to Juniper took the concept with them and implemented it in JunOS, so JunOS has this feature but IOS doesn't.

This article is part of You've asked for it series.

see 1 comments