Numerous articles describing the widespread routing instabilities caused by sloppy parser of a small router vendor (including posts at BGPmon, Renesys, Arbor Security and my blog) hinted that the unusual BGP update caused so many problems because the ISPs were using outdated Cisco IOS releases. This is definitely not the case; all classic IOS releases were affected.

Rodney Dunn from Cisco and myself were quickly able to reproduce so far unknown bug in Cisco IOS that occurs only when the inbound AS-path contains close to 255 AS numbers and the router does inbound or outbound AS-path prepending. The new bug is tracked as CSCsx73770 and affects downstream EBGP or IBGP sessions as follows:

The “BGP experiment” a small European ISP performed in February 2009 has generated quite a splash: Cisco has discovered a new BGP bug that can be triggered only if you have a long enough AS-path and do outbound AS-path prepending (and a few of us learned more BGP intricacies we never wanted to know), lots of people have (hopefully) discovered the importance of the bgp maxas-limit configuration command and at least some ISPs have implemented inbound prepending filters that I wrote about almost a year ago. However, most of us thought that the original problem arose due to inexperienced operators of a leaf AS.

In February 2009, a greenhorn ISP (they joined RIPE less than four months before the incident) in central Europe managed to generate a BGP update with too many AS numbers in the AS path, confusing older routers. You can find the details in an old Renesys blog post; at the peak of the instability, they were receiving over 100.000 BGP updates per second.

It’s very easy to protect yourself (and your downstream neighbors) from an operational error like this one. Cisco has implemented the AS-path length limiting code in IOS release 12.2. One would hope that the major ISPs would have started using this feature years ago; obviously that’s not the case, so here’s how you do it… just to make sure everyone understands what the bgp maxas-limit command does and hopefully implements it in this millennium.

Readers who commented on some of my previous certification-related posts have complained about the vagueness of exam questions. I have to agree with them; I’ve seen my fair share of dubious questions in the exams I’ve taken. For example, when I was developing EIGRP and BGP courses for Cisco, my lowest scores on the CCIE recertification exams were in those two categories. I knew too many details and was confused by the vagueness of the questions.

When I’ve stumbled across the headline Porn site feud spawns new DNS attack on NetworkWorld’s web site, the urge to read the article was simply irresistible. The article starts with the following paragraph (emphasis mine):

A scrap between two pornographic Web sites turned nasty when one figured out how to take down the other by exploiting a previously unknown quirk in the Internet's DNS.

The link in the paragraph points to another article documenting a completely different DNS attack. The next paragraph contradicts the first one (emphasis yet again mine):

The attack is known as DNS Amplification. It has been used sporadically since December, but it started getting talked about last month when ISPrime, a small New York ISP, started getting hit hard with what's known as a distributed denial of service (DDoS) attack.

Reading an interestingly-titled article on InformIT, I’ve stumbled across the following text:

The survival time is an estimate of how long an un-patched computer will remain uncompromised once it’s connected to the Internet. While the actual time varies, historically it tends to run between 4 and 20 minutes.

This is such an obvious nonsense that I had to check the source, which is also full of alarming messages, but admits at the end that the problems described largely disappeared with XP SP2. Just to put things in perspective: XP SP2 was released in August 2004 and the graph in the alarming blog post displays data from 2008.

Recently, on an IPSec-based customer network, we installed one of the brand new platforms introduced by Cisco Systems. The initial software release had memory leaks (no problem, we all know these things happen), so we upgraded the box to the latest software. It works perfectly … until you reload it. The software we’re forced to use cannot get IPSec to work if the startup configuration includes interface-level crypto-maps. Interestingly, you can configure crypto-maps manually and they work … until you save them into the startup configuration and reload the box.

What would you think if you’d receive three queries about the same (somewhat obscure) feature within six hours? It started with a nice e-mail from an engineer that I’ve corresponded with in the past. He wanted to send a Wake-on-LAN packet to a PC in a remote office. Usually you could use the ip directed-broadcast feature, but he wanted to use the remote office router to generate the packet.

Readers of my blog have probably noticed that I’m occasionally documenting the shortcomings of DNS and DHCP servers built into Cisco IOS (I will not even mention the HTTP server, this one gets constantly degraded). On the other hand, although you could centralize all these services, the centralization makes the branch offices completely dependent on the availability of WAN uplinks; without a working uplink, a branch office stops completely.

Another undocumented (and thus very probably unsupported) Tcl-on-IOS detail: numerous Tcl packages are bundled with IOS and available in the tmpsys:lib/tcl directory (the tmpsys: is a virtual file system mapped to a part of the IOS image).