Blog Posts in November 2008
I received an interesting comment on one of my knowledge/certification-related posts:
I used to think that certifications were a useful indicator of knowledge or at least initiative, but I’m changing my mind. [...] I feel like I’ve gotten a lot out of studying for certifications, especially CCIE, but I’m starting to wonder if that’s the exception.
I guess a lot of prospective internetworking engineers are thinking along the same lines, so here’s my personal perspective on this issue.
The Network World recently published a story describing the results of an independent security product testing lab, where they’ve discovered (surprise, surprise) that adding security features to Cisco routers “presents a tremendous bottleneck” and “can turn a 60G router into a 5G one or even a 100M bit/sec device”.
The test results haven’t been published yet; I’ve got all the quotes from the NW story, so they might be the result of an ambitious middleware.
We don’t need “independent experts” for that. Anyone who has ever configured VPNs in a high-speed environment can tell you how to kill the performance. The basics are always the same: make sure the dedicated silicon can’t handle the job, so the packets have to be passed to the CPU. Here are a few ideas:
A member of NIL’s forums wanted to run IS-IS over a hub-and-spoke Frame Relay network without using subinterfaces. I hope the question is not related to a production network; running IS-IS over a generic partially-meshed multi-access WAN network is not a good idea.
You might remember that the last polls I did using Blogger all resulted in every option having exactly the same number of votes. At that time, I've blamed Google ... and I have to apologize. It was obviously someone who has nothing better to do in his life. The log files I've collected indicate he's coming from Poland and I would appreciate if my Polish readers could help me persuade this troubled individual that he should spend his time doing something else (details in the rest of the post).
Let’s assume that you’re the manager of the internetworking team for a large enterprise network. You’ve just decided to migrate less-critical sites in your network from traditional (expensive) WAN offerings to IPSec running over the public Internet. Your internetworking architect has worked with the vendors to select the best technology and chose dynamic multipoint VPN (DMVPN) with a CA server running on a router. The proof-of-concept lab has been built and now you’re ready to order the new boxes and start the deployment. But there’s a major roadblock in this otherwise rosy scenario. Your engineers have to be trained on the new technology before the rollout; otherwise, you can expect interesting fallouts when the first problems inevitably start to appear.
The post describing my ideas about interactive online sessions resulted in a few comments and several off-line suggestions. Unfortunately most of the suggestions you’ve made in the comments are too generic. Remember, I was talking about 30-60 minute sessions and some suggestions would easily fill a week’s worth of training at the level of detail I’m aiming at. Running high-level introductory sessions is not my idea of fun; you could get as many of them as you want at Networkers.
Several suggestions are still “in the pipeline”: I have to envision how to structure them to make them manageable. In the meantime, the rest of the post lists the topics we can definitely cover. Please vote on them, the most popular one will be featured in December session.
I was delighted when I got access to Cisco's ACE XML Gateway/Web Application Firewall (WAF) box. This box is the perfect intersection of three fields I'm really interested in: networking, security and web programming, so I'll work with it quite a lot in the future and post interesting tips and tricks about its usage.
As this blog is currently focused exclusive on Cisco IOS, I'm wondering how to cover these new products. I won't create another blog; it simply doesn't make sense to build another blog from the ground up, but there are a few other options. Please help me select the best one by voting in the poll.
The November Technical Services News from Cisco included the Annotating Troubleshooting Sessions document from the Cisco’s support wiki. The document describes two well hidden features of Cisco IOS:
- The send log exec-level command writes a line in the syslog, allowing you to delineate logging or debugging outputs.
- The exclamation mark used as the first character in any IOS command line (not just in the configuration) serves as a comment. If you’re logging the TTY session, you can use these comments to document the session.
When I chose the word “unfortunately” in my post describing how Cisco IOS performs DNS lookup when you enter a host name in an access list, I’ve triggered several responses that disagreed with my choice of words. Here’s why I still think IOS ACL could be improved with dynamic DNS lookup:
I “always knew” that Cisco IOS supports NAT translations between local and global addresses in DNS replies … until I wanted to use this functionality in one of my sample configurations and discovered it doesn’t work as expected.
A few tests later, I discovered the true story: DNS requests and responses are translated if and only if you define IP-level NAT translations using either the ip nat inside source static or the ip nat inside source list pool configuration command. The translations should not use any additional filters (do not use the route-map keyword) and cannot result in PAT translations (do not use the overload keyword).
When I was configuring the access list that should prevent spammers from misusing my workstations, I obviously had to figure out the IP address of the ISP’s SMTP server (access lists and object groups accept IP addresses). I almost started nslookup on my Linux workstation, but then decided to try entering a hostname in an IOS ACL … and it works. Unfortunately, IOS performs a DNS lookup when you enter the hostname (assuming you have configured the ip name-server) and stores the resulting IP address in the ACL definition:
rtr(config)#ip access-list extended InsideList
rtr(config-ext-nacl)#permit tcp any host smtp.example.com eq smtp
Translating "smtp.example.com"...domain server (192.168.0.1) [OK]
rtr(config-ext-nacl)#do show access-list InsideList
Extended IP access list InsideList
10 permit tcp any host 192.168.2.3 eq smtp
You can enter hostnames in ACLs or network object groups. In both cases, the name is immediately translated into an IP address.
We’ve spotted some of our best engineers when they were in the final years of their undergraduate studies. To continue the trend, NIL offers a student-engagement program that attracts highly promising candidates each year. They offer them CCNA training (after which the students have to pass the exam), a few weeks of hands-on instructor-led introductory bootcamps and the first CCNP course. These training courses should give students a solid foundation and a framework that they can expand on their own—which is the point where it's time to stress-test them with advanced bootcamps.
My MPLS QoS: Implementing the best model for guaranteed service article published by SearchTelecom gives you a high-level overview of the pipe and hose QoS models in the MPLS VPN environment. I’m also describing basic DiffServ QoS mechanisms available in an MPLS backbone.
If you’re new to IP QoS, you should start with the IP QoS: Two generations of class-of-service tools article.
In mid-December, I’ll do my first IOS Hints Online Session. These sessions will be short (30-60 minutes), very interactive (I hope, but that’s your choice) and focused on an interesting design/deployment aspect. The description of the design/deployment challenge addressed by the session will be available well in advance at the time when you’ll be able to register.
Each session will start with a few diagrams explaining the proposed solution to the session’s topic and continue with hands-on explanation on actual devices. Each session will be limited to ~15 participants who will be able to actively participate, ask questions, propose alternative solutions or even discuss their actual issues (assuming they are somewhat related to the primary topic of the session).
I have a “few” ideas what could be covered in these sessions, but having a real-life challenge coming from the readers of my blog would be much better. If you have a good idea that could fit into this concept, please send me a short description before Friday, November 21st. I’ll collect the best ones, publish short descriptions in a blog post and you’ll prioritize which ones you’d like to see first.
I always thought that there was no need to restrict outbound sessions across a firewall in low-security environments. My last encounter with malware has taught me otherwise; sometimes we need to protect the rest of the Internet from our clumsiness. OK, so I decided to install an inbound access-list on the inside interface of my SOHO router that will block all SMTP traffic not sent to a well-known SMTP server (and let the ISP’s SMTP server deal with relay issues).
Control plane (the main CPU that runs the routing protocols and all other application-layer services) is the most vulnerable part of your router. A determined attacker can quickly overload the CPU of any router (or switch) with a targeted denial-of-service attack, either by sending IP packets that are propagated from the switching fabric (or interrupt code on software-only platforms) to the control plane processes or by targeting individual services running on the router (see, for example, the problems one of the readers had with public DNS server running on the router).
Cisco IOS offers several control plane protection mechanisms. I’ve summarized them in the “Protecting the router’s control plane” article in the CT3 wiki and Sebastian Majewski has provided sample router configuration.
Reading the stories of Windows workstations becoming members of a spam botnet becomes way less enjoyable when you’re faced with the same problem (one of my kids managed to install a Trojan). It took me a day to clean the infected computer (it would have been easier to just format it, but the repeated installation of the Windows XP + Office software is so boring), but I’ve learned a few interesting networking lessons in the process that I’ll document in the next days.
I was very anxious to get my copy of Cisco Secure Firewall Services Module (FWSM) from Cisco Press, as I’m a purely router-focused person, and I wanted to understand the capabilities of the Firewall Services Module (PIX/ASA-like blade for the Catalyst 6500 switching system with virtual firewall capability). I have a good background in IOS-based firewalls and network address translation (NAT), so the book was a perfect fit for me. However, if you’re looking for “best practices for securing networks with FWSM,” you’ve been misled by the subtitle.
One of my kids managed to get infected with a particularly sneaky Facebook Trojan: a link from a friend (probably also infected) pointed to a web page with a video that required installation of a newer version of the Flash player … which was actually the first part of the Trojan. It quickly downloaded a few more components and made itself cozy deep within Windows XP.
Before you start telling me that kids would click anything … we had “a few” not so very pleasant discussion after previous infections and they know not to open anything or click on something that looks strange. Unfortunately the update-happy industry has conditioned them to constant prompts to upgrade one or another component and the request to upgrade the Flash player was obviously too legitimate-looking.
As always, Jeremy Stretch posted several interesting articles: how to hijack HSRP, introduction to split horizon in distance vector routing protocols and (long needed) default redistribution metrics.
Petr Lapukhov started playing with HTTP URL regular expressions within NBAR and documented his findings. The most interesting is the last Q/A pair: can I use NBAR as a content filtering engine?
And last but definitely not least, if you’re worried what will happen to WPA2 now that WPA has been cracked, Robert Graham explains the fundamental differences between WPA and WPA2. Also, make sure you read the detailed explanation of the WPA flaw to understand its implications.
BFD is one of those simple ingenious ideas that make you wonder “Why did it take them so long to figure this out?” It’s a UDP-based protocol that replaces dozens of link-level failure-detection mechanisms and routing protocol tweaks with a simple, focused solution: detect hop-by-hop layer-3 failures.
I wanted to write about BFD a year ago when it was first advertised as being available in the low-end routers (BFD support on high-end platforms is much better, but I simply don’t have a GSR and a CRS-1 at home … yet), but it failed to work, so I had to shelve the idea until the IOS release 12.4(15)T matured to a point where BFD on ISR started working in IOS, not just in Powerpoint.
In this month’s IP corner article, “Improve the Convergence of Mission-Critical Networks with Bidirectional Forwarding Detection (BFD)”, I’m describing BFD principles, its configuration on Cisco IOS and give you practical examples how you can use BFD to improve next-hop failure detection.
Quagga is a terrifically useful tool when you need to build a BGP test lab. Not only can you quickly add an extra BGP router in your network; it also allows you to insert BGP routes with almost any attribute you want. I’ve described some of its features and included a sample Quagga-to-router connectivity scenario in the “Use Quagga to generate BGP routes” article published in the CT3 wiki.
Those readers that have been discussing technical issues with me probably know that I rarely write something without testing it first. Somehow I didn’t feel like powering up our spare CRS, so you might wonder how I’ve tested the interoperability between four-byte AS implementations and Cisco IOS. Fortunately, there’s open-source routing protocol software suite named Quagga (which is an extinct subspecies of zebra in the real world) that has already implemented the new BGP standards and allowed me to do all the tests with just a router and a Linux host.
To help you get started, I wrote an article in the CT3 wiki describing the Quagga installation and configuration process on Fedora Linux.
Quagga is also available as binary package (RPM) for Red Hat/CentOS/Fedora, Solaris, Debian and Gentoo, but you'll most probably get at least a year old version. Vitaliy Gladkevitch provided RPM installation instructions.
I’ve received almost a dozen responses to the second OSPF challenge, most of them correct. The key to the solution is the way OSPF checks neighbor’s IP address on point-to-point links (we already know that the subnet mask is ignored):
- If the interface is unnumbered, the router ignores the source IP address in the OSPF hello packets.
- If there’s an IP address configured on the interface, the router checks that the neighbor’s IP address (the source IP address in the OSPF hello packets) belongs to the same subnet. If the source IP address is not in the same subnet, the OSPF hello packet is ignored.
In the “internet meltdown” post I’ve described the main reason for the routing problems we’re experiencing in the Internet: everyone wants to be truly multihomed. All these end-customers obviously need their own AS number and it’s no wonder the experts predict we’ll run out of AS numbers in two to three years.
There’s no need to panic: the technical solution (four byte AS numbers) has been ready for several years … but it’s not implemented yet in majority of Cisco IOS-based platforms. Does that mean we’ll experience Internet-wide problems when the regional registries start allocating AS numbers larger than 65536 in a few months? Luckily, the answer is NO, the new BGP standards are completely backward-compatible … but if you’re a Service Provider, you have to start thinking about the upgrade path.
You can find more answers on this topic
in the article I wrote for SearchTelecom.
This post brought back some ancient memories … and I’m always amazed how far we’ve got in the last 30 years. For me, it all started with an IBM 360, having 48K (forty eight kilobytes) of core memory in which it ran an operating system and three user partitions. Fortran IV was the only programming language and card reader the only input device.
Moving to a VAX 11/780 was a major improvement; it was a multitasking environment with real terminals. VAX was an interesting beast: the first step in the boot process was to start an embedded PDP-11 processor that read an 8” floppy disc and uploaded the microcode to the main CPU. The only drawback was that 30 users had to share 2M (two megabytes) of main memory and so I couldn’t crash the machine whenever I wanted.
A few years later, I managed to get access to a really cute research PDP-11 running RSX-11M. Finally I could start writing device drivers and kernel code without risking the wrath of dozens of users years older than myself. And then the personal computers appeared and I probably made one the best choices I could – the BBC Micro from Acorn. It was never popular, but it had an amazingly well-designed operating system that you could extend in any way you wish (and even symbolic assembly language built into its BBC BASIC).