Building Network Automation Solutions
6 week online course starting in September 2017

Basic MPLS VPN design & configuration guidelines

I’ve collected very basic MPLS VPN design and configuration guidelines and squeezed them into a two page PDF document. If you’re new to MPLS VPN, you’ll probably find it useful … although reading the MPLS and VPN Architectures book still remains the mandatory next step.

Download the document from the CT3 wiki

Is it wise to use certifications in the candidate selection process?

My previous certification-related post described how some companies use certifications to filter job applicants for networking-related positions. Should you follow that example? If you’re in a country with a saturated job market, where the number of applicants far exceeds the number of job postings (consider yourself very lucky if you’re an employer), you should certainly use whatever filters you can to screen the hundreds of applications you receive … but be aware that you have potentially lost a few gems hidden in the flood.

MPLS support on 1800-series routers

Christoph sent me an interesting question a few days ago:

I played a bit arround with 2 Cisco 1803 and I found MPLS related configurations commands in IOS 12.4(15)T (Advanced Enterprise) on this box. MPLS was not listed as a included fearture in the Cisco Feature Navigator for this image and some searching at took me to a 2 year old document telling me that MPLS isn't supported on this series. Some more searching took me back to the Cisco Feature Navigator which lists MPLS as feature for the Cisco 1805 router (which uses the same IOS image, afaik).
So, I'm a bit confused now if MPLS is really working / supported on the low-end Cisco ISR 1800 fixed series?

MPLS was mostly available but never supported on low-end platforms (including Cisco 2600). In those days I've taken some heat for reusing existing 2600-based labs to teach Cisco-internal MPLS courses (since we were teaching the students to configure unsupported devices :).

Anyhow, the "not supported" means exactly that: it may be available (well, it is), it may work (it actually does), but if it's broken (and I've seen at least one low-end-platform-specific bug in the early days) you can't complain.

Is anyone aware whether the official support for the MPLS on 1800 series has changed? If so, please share your information with us.

If you need to offer a production-grade service to your customers, don't use unsupported equipment; if you need a solution for your personal needs or you're building a lab, go ahead.

New in CT3 wiki: IP SLA Monitoring

Jeremy Stretch was kind enough to publish his IP SLA monitoring article in the CT3 wiki, at the same time changing the command syntax from the ip rtr command set to the newer ip sla command set. Thanks, Stretch!

Read the article in the CT3 wiki

Why would I need a Web Application Firewall?

If you have been visited by a friendly Cisco sales engineer recently, you might have already heard about the ACE Web Application Firewall (WAF). If you’re curious enough to start investigating on your own, you might have stumbled across the WAF product description on Cisco’s Web site, which tells you … nothing.

Let’s start with an easy question: if I already have a firewall, why would I need another box with “fire” and “wall” in its name? The short answer is “Because Web programmers rarely know how to write secure Web applications.”

Correction: NAT-translated DNS responses are not cacheable

It looks like the wording in the “NAT-translated DNS responses are not cacheable” post was a bit too vague, as some readers understood the router would mess the TTL field in the DNS response payload when changing the IP addresses in the IP header of the response packet.

That's not the case; the TTL field in the DNS response payload is touched only if the router performs application-layer translation of the DNS response (for example, changing the A record in the DNS response). I've reworded the original post; I can only hope I've made it unambiguous (after all, English is not my native language).

MPLS VPN terminology

I put together a short list of MPLS VPN-related terms; I'm positive you'll find it handy if you've recently entered this exciting technology area. With copious help from our marketing department I was also able to produce a PDF version of the same document that you can download from the CT3 wiki.

Read more in the CT3 wiki

NAT-translated DNS responses are not cacheable

Slightly reworded on 2008-12-18. Not only is NAT (as implemented in Cisco IOS) very picky about the translation of IP addresses in the payload of DNS responses (it translates only addresses defined in IP-level NAT translations with no additional route-map filters), it also sets the TTL field in the DNS response to zero when translating an IP address in the DNS response payload based on an existing L3 NAT entry making the DNS response completely uncacheable.

The behavior makes some sense, as the L3 NAT entry might change before the DNS response expires, but the implementation is definitely overly aggressive. In my opinion, IOS should use some sensible (configurable?) value for static NAT translations and times comparable to NAT timeouts for dynamic NAT translations. What do you think?

Designing Site-to-Site IPsec VPNs – Part 3

In the third part of the Site-to-Site IPsec VPN series in our IP Corner, Boštjan Šuštar describes the Virtual Tunnel Interfaces (VTI) and the ways this relatively new addition to Cisco IOS can be mixed with crypto maps and Easy VPN solution.

Book review: Voice over IP Security

Based on the title, I would assume that the Cisco Press book Voice over IP Security: Security best practices derived from deep analysis of the latest VoIP network threats attracts primarily senior voice engineers who know that they have to secure their production networks. The author of the book strongly disagrees with my opinion, however, spending more than a third of the book on baseline explanations of VoIP, SIP, H.323, firewalls, NAT, DES, IPSec…. I enjoyed the overview chapters, as I last configured VoIP before SIP was invented, but an experienced VoIP engineer would be disappointed.

This is QoS; who cares about real-time response?

It all started with a innocuous question: can you detect voice traffic with EEM? Looks simple enough: create a QoS class-map that matches voice calls and read the cbQosClassMapStats table in the CISCO-CLASS-BASED-QOS-MIB. The first obstacle was finding the correct indexes, but a Tcl script quickly solved that; I was ready to create the EEM applet. The applet failed to work correctly and after lots of debugging I figured out the counters in the cbQosClassMapStats table change only every 10 seconds.

I couldn’t believe my eyes and simply had to test other MIB variables as well. As expected, the IF-MIB (standard interface MIB) counters increase in real-time, but obviously someone had the bright idea that we need to detect changes in traffic profile only every now and then. Although I've received suggestions from my readers, none of them works on an 1800 or a 7200. Oh, well, Cisco developers from the days when I started working with routers would know better.

To test the MIB variable behavior I wrote a simple Tcl script to test the MIB variables. It reads the specified MIB variable at fixed intervals and prints the values, so you can monitor the changes in the MIB variable in real-time. I started low-bandwidth UDP flood across the router and monitored the output bytes interface counter. As expected the counter changed in real time and accurately tracked the amount of traffic sent through the router.

GW#pm ifOutOctets.3 public 10 1000          
polling ifOutOctets.3 for 10 seconds (10 iterations)

 0.000 ifOutOctets.3=42528679        
 1.000 ifOutOctets.3=42537767        
 2.000 ifOutOctets.3=42546713        
 3.000 ifOutOctets.3=42555719        
 4.000 ifOutOctets.3=42564665        
 5.000 ifOutOctets.3=42573611        
 6.000 ifOutOctets.3=42582699        
 7.000 ifOutOctets.3=42591645        
 8.000 ifOutOctets.3=42600591        
 9.000 ifOutOctets.3=42609537        

Then I created a simple class-map and policy map

GW#show policy-map interface

  Service-policy output: LAN

    Class-map: Voice (match-all)
      20438 packets, 2902196 bytes
      30 second offered rate 70000 bps
      Match: access-group name Voice

    Class-map: class-default (match-any)
      41 packets, 3967 bytes
      30 second offered rate 0 bps, drop rate 0 bps
      Match: any 

… and monitored the pre-policy byte counter (cbQosCMPrePolicyByte64) for the Voice class. The value changed only once every ten seconds:

GW#pm cbQosCMPrePolicyByte64.50.10767521 public 10 1000
polling cbQosCMPrePolicyByte64.50.10767521 for 10 seconds (10 iterations)

 0.000 cbQosCMPrePolicyByte64.50.10767521=0x002865366
 1.000 cbQosCMPrePolicyByte64.50.10767521=0x002865366
 2.000 cbQosCMPrePolicyByte64.50.10767521=0x002865366
 3.000 cbQosCMPrePolicyByte64.50.10767521=0x002865366
 4.000 cbQosCMPrePolicyByte64.50.10767521=0x002865366
 5.000 cbQosCMPrePolicyByte64.50.10767521=0x002865366
 6.000 cbQosCMPrePolicyByte64.50.10767521=0x00287acf8
 7.000 cbQosCMPrePolicyByte64.50.10767521=0x00287acf8
 8.000 cbQosCMPrePolicyByte64.50.10767521=0x00287acf8
 9.000 cbQosCMPrePolicyByte64.50.10767521=0x00287acf8

What can I say ... apart from expressing my deepest disappointment :(

The most convoluted MIB I’ve seen

Jared Valentine sent me a really interesting problem: he would like to detect voice traffic and start shaping TCP traffic for the duration of the voice call. The ideal solution would be an EEM applet reacting to the changes in the CISCO-CLASS-BASED-QOS-MIB; one of its tables contains the amount of traffic for each class configured in a service policy.

The MIB navigation looks simple: you just read the values from the cbQosClassMapStats table, indexed by policy ID and class ID. The real problem is finding the correct index values. I could walk the MIB manually with a MIB browser or snmp_getnext TCL calls, but this approach is obviously not scalable, so I wrote a script that walks through the cbQosServicePolicy, cbQosObjects, cbQosPolicyMapCfg and cbQosClassMapCfg tables and prints the index values you need.

You can find more Tcl-related information in the Tclsh on Cisco IOS tutorial. Sample Tclsh scripts are available in the Tclsh script library. If you need expert help in planning, developing or deploying Tclsh scripts in your network, contact the author.

Cisco AXP Developer Contest

Do you have a great idea what a router should be able to do, but you cannot make it work because you can't tweak router's embedded management tools (including Tcl)? Here is an opportunity you shouldn't miss: enroll in the Cisco AXP Developer Contest. Even if you're absolutely positive that having an extra Linux server sitting next to the router is better than a router blade (and some Linux pundits had lots to say on the topic), sending your ideas to Cisco won't cost you more than a few moments of your time and an e-mail ;)

And, BTW, mundane (but nonetheless highly useful) ideas like »visual policy map editor« or truly integrated DNS/DHCP server probably won't get you very far (that's why I'm not applying :).

Full disclosure: I was kindly asked if I could help spreading the word about the contest, but all the enthusiasm is exclusively mine.

Certifications: a new barrier to entry

Recent blog posts indicate that, in at least some market segments, IT certifications are becoming a new barrier to entry: companies require a specific set of certifications in their job offerings and use those requirements to filter the candidates who are invited to the initial interview. Obviously, IT vendors pushing the certifications are getting some real traction. On the other hand, anecdotal evidence indicates that certification holders are sometimes able to memorize vast amounts of information without being able to put it to use (I don’t want to imply that they used other, less honest methods).

NAT translation logging

The ip nat log translations syslog command starts NAT logging: every NAT translation created on the router is logged in syslog (which can be sent to console, syslog host or internal buffer). You could use this command as a poor man’s reporting tool if you have to monitor the address translations on your edge routers (for example, due to security policy or auditing requirements). Obviously you should configure the no logging console first in a production environment; otherwise your router will hang a few moments after you’ve enabled NAT logging.

The logging printouts include:

  • layer-4 protocol;
  • inside local and global addresses and port numbers;
  • outside local and global addresses and port numbers.

For example, the following messages were logged when an inside host tried to access web server at the IP address The inside source address was translated through a NAT pool (configured with ip nat inside source list ACL pool POOL command) into and the inside destination address was translated into outside destination address (configured with ip nat outside source static command).

12:17:12.503: %IPNAT-6-NAT_CREATED: Created tcp
12:18:47.751: %IPNAT-6-NAT_DELETED: Deleted tcp

Does it sound weird to specify the destination address translation with the ip nat outside source configuration command? It does to me …

Online session poll results

I would like to thank everyone that took time and voted on the subject of my next on-line session. We have a clear winner: Dynamic routing across a firewall … although the closeness of the other topics might indicate a helping hand of our polish friend. Unfortunately, I have some bad news to go with the good ones – the “December” session will most likely happen in January or February (I'm simply running out of time).

Update interval for IOS MIB counters?

Dear readers! This time I really need your help (uncle Google and his relatives gave me only one relevant hit and even that doesn't work on 7200 or 1800).

I'm trying to implement an EEM applet that would detect traffic rate change using CISCO-CLASS-BASED-QOS-MIB. Everything would work perfectly ... if only IOS wouldn't update the MIB counters approximately every 10 seconds, not in real-time. Is anyone aware of a configuration command that would force the router to update these counters any faster?

Random “Scenic Route Certification” thoughts

The “Sometimes the path is more important than the destination” post has generated numerous highly interesting comments. I already planned to write about some of the issues raised by the readers (certification grind mill) or wrote about others (knowledge or recipes), so I’ll skip those and focus on the other interesting bits-and-pieces (but please make sure you read the original post first).

Fix the EEM RFC 822-related bug

A while ago I’ve reported that Cisco IOS does not conform to RFC 822, potentially resulting in ill-formed e-mails send from EEM applets. The bug has been fixed in IOS releases 12.4(15)T2 (and all later 12.4T releases) and 12.2(33)SRC, but if you’re stuck with an older release, you can use a nice trick Gerald Kowalsky sent me.

Read more in CT3 wiki

Should VTP be disabled by default?

One of my readers sent me a question that triggered one of my old grudges:

In my experience, when you first add a new switch (having a NULL domain) on an existing VTP Domain, it inherits the domain name, regardless of it being a VTP Server. I was wondering if this is a feature (i.e. has proved to be a solution in most cases) or a bug (i.e. has proved to cause problems in most cases). I know it's proved to be the latter for us!

In my personal opinion Cisco at one point in time wanted too much plug-and-play and someone had a great idea that you can just plug another switch into your network and it would autoconfigure itself. We've been suffering because of that "insight" ever since (and the CCIE written test has material for a few more interesting questions :).

I strongly believe that VTP should be turned off by default and should generate a warning before being enabled, but it will probably not happen. What do you think?

Disclaimer: I am not a switching person and have no idea about anything below or above layer 3.

Configuring IS-IS on partially-meshed multi-access WAN network

If someone insists on running IS-IS on a partially-meshed Frame Relay network without using subinterfaces (which is the proper way of doing it), it can still be done … assuming you understand what’s going on and properly configure the IS-IS routers.

Read the configuration guidelines in the CT3 wiki.

I’m too old … I prefer CLI over GUI

I was delighted when I got access to Cisco’s Application Control Engine (ACE) XML Gateway/Web Application Firewall (WAF) box. This box is the perfect intersection of three fields that really interest me: networking, security and Web programming. To my huge disappointment, though, all the real configuration can only be done through the Web interface. I understand that casual users of a device prefer a graphical user interface (GUI) over text commands (and Generation Z has never seen a terminal window, DOS prompt or, God forbid, an actual terminal), but you can achieve so much more with a simple text-based configuration approach:

ACE XML Gateway Poll Results

Thanks to everyone who took time to participate in my “ACE XML Gateway” poll (and special thanks to our anonymous friend from Poland who amused numerous readers and helped me set up slightly more secure polls). A large majority (with the notable and vocal exception of the previously-mentioned anonymous contributor) would like to see posts (or at least pointers to them) in the IOS Hints blog:

When would an MPLS LSR have Untagged output label?

This is a nice MPLS question I’ve received from one of the readers:

I have understood the Penultimate Hop Popping (PHP) process, but I don’t understand when a router would use UNTAGGED instead of POP TAG?

Instead of answering the question directly, let's walk through a series of simple Q&A pairs that will help you understand the whole process (remember: knowledge, not recipes!).

It's highly recommended you read the first few chapters of the MPLS and VPN Architectures book before the rest of this post.

Where does the Untagged keyword appear? It only appears as the output label in the LFIB (Label Forwarding Information Base) that you can inspect with the show mpls forwarding-table.

What does the Untagged keyword mean? This keyword means that the router has no output label associated with the forwarding equivalence class (FEC ... usually an IP prefix). Since there is no output label, the router cannot perform a label swap (or pop) but has to remove the whole MPLS shim header.

Where would a router get the output label? It's received from the next-hop router.

When would a router have no output label? When there is no next-hop router or when the next-hop router did not advertise a label for the IP prefix.

When would there be no next-hop router? If the IP prefix is a directly connected subnet (including a loopback interface) or a summary route advertised by the router itself.

When would the next-hop router not advertise a label? The reasons a next-hop router would not advertise a label for an IP prefix include:

  • It's not running MPLS.
  • It's running MPLS but not CEF (MPLS labels are assigned to IP prefixes in CEF table).
  • It's not reachable across an MPLS-enabled interface (both routers could be running MPLS, but the transit interface does not have the mpls ip configuration).
  • The LDP session has not been established yet.
  • There is a mismatch in LDP protocol (one router is running Cisco's proprietary TDP, the other one standard LDP).
  • The next-hop router uses an access-list to filter the IP prefixes for which the MPLS labels are advertised.

Summary: you would see the Untagged label in the LFIB when the IP prefix is a directly connected interface, a summary route or the next-hop router has not advertised the label.

Sometimes the path is more important than the destination ...

I received an interesting comment on one of my knowledge/certification-related posts:

I used to think that certifications were a useful indicator of knowledge or at least initiative, but I’m changing my mind. [...] I feel like I’ve gotten a lot out of studying for certifications, especially CCIE, but I’m starting to wonder if that’s the exception.

I guess a lot of prospective internetworking engineers are thinking along the same lines, so here’s my personal perspective on this issue.

This is why I don’t trust “independent experts”

The Network World recently published a story describing the results of an independent security product testing lab, where they’ve discovered (surprise, surprise) that adding security features to Cisco routers “presents a tremendous bottleneck” and “can turn a 60G router into a 5G one or even a 100M bit/sec device”.

The test results haven’t been published yet; I’ve got all the quotes from the NW story, so they might be the result of an ambitious middleware.

We don’t need “independent experts” for that. Anyone who has ever configured VPNs in a high-speed environment can tell you how to kill the performance. The basics are always the same: make sure the dedicated silicon can’t handle the job, so the packets have to be passed to the CPU. Here are a few ideas:

  • Configure GRE over IPSec and make sure you don’t tweak the MTU on the GRE tunnel. This will result in IP fragmentation and the receiving router will have to process every fragment in process switching path. A sure killer for any box, not just the 6500/7600.
  • Make sure you configure features for which you have no hardware accelerator installed in the high-end boxes and watch the performance fall (at least) 100x.
  • Even if you’ve managed to install an accelerator, configure the network in a way that effectively disables the hardware. For example, configure multiple GRE tunnels terminating on the same loopback interface
  • Design your test so that all the traffic has to pass through a bottleneck. FWSM with its 3-5GBps throughput is an ideal candidate.

What these tests prove to me is that someone who doesn’t understand what he’s doing can destroy the performance of almost any device … but we don’t need independent tests to prove that. Am I missing something? Please let me know.

IS-IS over partially meshed Frame Relay

A member of NIL’s forums wanted to run IS-IS over a hub-and-spoke Frame Relay network without using subinterfaces. I hope the question is not related to a production network; running IS-IS over a generic partially-meshed multi-access WAN network is not a good idea.

More details are available in the CT3 wiki.

Stuffing the polls: the adventures of a convoluted mind

You might remember that the last polls I did using Blogger all resulted in every option having exactly the same number of votes. At that time, I've blamed Google ... and I have to apologize. It was obviously someone who has nothing better to do in his life. The log files I've collected indicate he's coming from Poland and I would appreciate if my Polish readers could help me persuade this troubled individual that he should spend his time doing something else (details in the rest of the post).

I've decided to use another polling service for the current set of polls, just to make sure it was not a Blogger problem. Polls went smoothly and displayed an expected spread of votes, but yesterday morning I've noticed that the number of votes for each option were getting more and more equal. Fortunately, the new polling service allows me to track votes by IP address, so I was quickly able to discover that someone using the IP address was stuffing the ballot box. I've cleared the votes and hoped he'd realize he's been discovered and stop.

Well, this individual realized he's been discovered ... and moved over to a proxy server belonging to a system integrator (; and a private DSL connection (; By this morning he submitted over 600 votes and now he moved back to (that could be where he works, as the IP address is just one hop away from the POS interface of a Telenergo router).

As said above, anything you can do to help me would be much appreciated (I would prefer this over writing complaining e-mails to the postmaster and abuse aliases of the affected networks). Thanks!

Tailor the certification training to your needs

Let’s assume that you’re the manager of the internetworking team for a large enterprise network. You’ve just decided to migrate less-critical sites in your network from traditional (expensive) WAN offerings to IPSec running over the public Internet. Your internetworking architect has worked with the vendors to select the best technology and chose dynamic multipoint VPN (DMVPN) with a CA server running on a router. The proof-of-concept lab has been built and now you’re ready to order the new boxes and start the deployment. But there’s a major roadblock in this otherwise rosy scenario. Your engineers have to be trained on the new technology before the rollout; otherwise, you can expect interesting fallouts when the first problems inevitably start to appear.

An offer you should not refuse

Just stumbled across this: Amazon is offering the MPLS and VPN Architectures, MPLS and VPN Architectures, Volume II and Internet Routing Architectures (2nd Edition) (Networking Technology) (from Sam Halabi) for a total of $160.

Online sessions in December 2008: please vote!

The post describing my ideas about interactive online sessions resulted in a few comments and several off-line suggestions. Unfortunately most of the suggestions you’ve made in the comments are too generic. Remember, I was talking about 30-60 minute sessions and some suggestions would easily fill a week’s worth of training at the level of detail I’m aiming at. Running high-level introductory sessions is not my idea of fun; you could get as many of them as you want at Networkers.

Several suggestions are still “in the pipeline”: I have to envision how to structure them to make them manageable. In the meantime, the rest of the post lists the topics we can definitely cover. Please vote on them, the most popular one will be featured in December session.

Building a transit autonomous system with no BGP in the core

This idea came from the discussion in the CCIE Journey blog: how do I pass packets across a network that does not run BGP on every router (for example, from X1 to X2 in the following diagram). The solution in the CCIE Journey blog used GRE tunnels between edge routers, we’ll use MPLS.

Dynamic routing across a firewall

This topic started as a simple question: “How can I achieve dynamic failover to disaster recovery site if my security engineer refuses to configure dynamic routing on the firewall”. We’ll solve the problem in a simple network shown in the following diagram:

Reducing the size of the BGP table

Anyone who uses a hardware-based layer-3 switching device (which is almost any high-speed router these days) for a core router could be hit by this problem: as the number of routable prefixes in the Internet increases, you might run out of hardware lookup entries (TCAM, for example). How do you reduce the size of the IP routing table without losing too much flexibility? What are the drawbacks and the caveats?

BGP Autonomous System split

What happens if your BGP autonomous system splits in half due to a link failure? Can you patch it together? What are the caveats?

How should I cover ACE XML Gateway and Web Application Firewall?

I was delighted when I got access to Cisco's ACE XML Gateway/Web Application Firewall (WAF) box. This box is the perfect intersection of three fields I'm really interested in: networking, security and web programming, so I'll work with it quite a lot in the future and post interesting tips and tricks about its usage.

As this blog is currently focused exclusive on Cisco IOS, I'm wondering how to cover these new products. I won't create another blog; it simply doesn't make sense to build another blog from the ground up, but there are a few other options. Please help me select the best one by voting in the poll.

Annotate your router sessions

The November Technical Services News from Cisco included the Annotating Troubleshooting Sessions document from the Cisco’s support wiki. The document describes two well hidden features of Cisco IOS:

  • The send log exec-level command writes a line in the syslog, allowing you to delineate logging or debugging outputs.
  • The exclamation mark used as the first character in any IOS command line (not just in the configuration) serves as a comment. If you’re logging the TTY session, you can use these comments to document the session.

3 reasons why I would like to have DNS lookups in IOS access lists

When I chose the word “unfortunately” in my post describing how Cisco IOS performs DNS lookup when you enter a host name in an access list, I’ve triggered several responses that disagreed with my choice of words. Here’s why I still think IOS ACL could be improved with dynamic DNS lookup:

  • Things change. If you have to match a specific host in your ACL, there’s no guarantee that the host’s IP address will stay the same indefinitely. If the host is within your network and your ACL breaks because the host’s IP address was changed, it’s your problem (you should have kept better documentation and implemented proper change management procedures). When you have to use an external IP address (for example, the ISP’s SMTP gateway), you’ll notice it has changed when the phones start to ring.
  • Self-documentation. If the hostnames would remain in the ACLs and the router would perform a lookup as needed, the access lists would be self-documenting. When the hostnames get replaced by IP addresses, you have to perform reverse lookup manually to figure out what host the IP address is referring to.

You could use remark commands in access-lists to document what you’re doing. Although you can use multiple remark commands in the same ACL, they cannot be edited like the filtering lines in the ACL.

  • Reverse lookup problems. The IP address entered in the ACL does not necessarily translate back into the host name you’ve used. In some cases (hosted applications), the reverse lookup might give you a host name in a completely different domain, making your deciphering job even harder (assuming, of course, that your predecessor left no documentation behind).

There are, of course, numerous minor issues that would need to be addressed, for example:

  • Load balancing. Properly implemented DNS-based load balancers return numerous randomly mixed IP addresses as a response to the A query. The IOS could convert multiple returned addresses into a network object group automatically.
  • TTL issues. In most cases, the DNS zone files contain meaningful TTL values (the IP addresses stay valid for minutes or hours). Even if the router performed the DNS lookup for every packet (which would be total nonsense), it would usually get the same results on every query due to a cache somewhere in the chain between the router and the final DNS server. The DNS lookup thus only makes sense when the DNS A record expires.
  • Short TTL issues. Sometimes the responses returned by the DNS server contain very low TTL values (TTL might also be set to zero to disable caching). In these cases, IOS could provide a minimum TTL parameter and warn the operator when a hostname is used that results in a response with TTL below the threshold.

In any case, the saddest part of the story is that the IOS already supports the same functionality in a different part of the code: dynamic DNS lookups are used in zone-based firewall policies to identify masquerading applications like MSN and Yahoo messenger (see Chapter 5 of the Deploying Zone-Based Firewalls digital book).

Network Address Translation of DNS responses

I “always knew” that Cisco IOS supports NAT translations between local and global addresses in DNS replies … until I wanted to use this functionality in one of my sample configurations and discovered it doesn’t work as expected.

A few tests later, I discovered the true story: DNS requests and responses are translated if and only if you define IP-level NAT translations using either the ip nat inside source static or the ip nat inside source list pool configuration command. The translations should not use any additional filters (do not use the route-map keyword) and cannot result in PAT translations (do not use the overload keyword).

You can find more details in the “Network address translation of DNS responses” article in the CT3 wiki.

Using hostnames in IP access lists

When I was configuring the access list that should prevent spammers from misusing my workstations, I obviously had to figure out the IP address of the ISP’s SMTP server (access lists and object groups accept IP addresses). I almost started nslookup on my Linux workstation, but then decided to try entering a hostname in an IOS ACL … and it works. Unfortunately, IOS performs a DNS lookup when you enter the hostname (assuming you have configured the ip name-server) and stores the resulting IP address in the ACL definition:

rtr(config)#ip access-list extended InsideList
rtr(config-ext-nacl)#permit tcp any host eq smtp
Translating ""...domain server ( [OK]
rtr(config-ext-nacl)#do show access-list InsideList
Extended IP access list InsideList
    10 permit tcp any host eq smtp

You can enter hostnames in ACLs or network object groups. In both cases, the name is immediately translated into an IP address.

The best way to learn: solve a hard challenge

We’ve spotted some of our best engineers when they were in the final years of their undergraduate studies. To continue the trend, NIL offers a student-engagement program that attracts highly promising candidates each year. They offer them CCNA training (after which the students have to pass the exam), a few weeks of hands-on instructor-led introductory bootcamps and the first CCNP course. These training courses should give students a solid foundation and a framework that they can expand on their own—which is the point where it's time to stress-test them with advanced bootcamps.

MPLS QoS: Implementing the best model for guaranteed service

My MPLS QoS: Implementing the best model for guaranteed service article published by SearchTelecom gives you a high-level overview of the pipe and hose QoS models in the MPLS VPN environment. I’m also describing basic DiffServ QoS mechanisms available in an MPLS backbone.

If you’re new to IP QoS, you should start with the IP QoS: Two generations of class-of-service tools article.

Interactive online sessions: your input is highly appreciated

In mid-December, I’ll do my first IOS Hints Online Session. These sessions will be short (30-60 minutes), very interactive (I hope, but that’s your choice) and focused on an interesting design/deployment aspect. The description of the design/deployment challenge addressed by the session will be available well in advance at the time when you’ll be able to register.

Each session will start with a few diagrams explaining the proposed solution to the session’s topic and continue with hands-on explanation on actual devices. Each session will be limited to ~15 participants who will be able to actively participate, ask questions, propose alternative solutions or even discuss their actual issues (assuming they are somewhat related to the primary topic of the session).

I have a “few” ideas what could be covered in these sessions, but having a real-life challenge coming from the readers of my blog would be much better. If you have a good idea that could fit into this concept, please send me a short description before Friday, November 21st. I’ll collect the best ones, publish short descriptions in a blog post and you’ll prioritize which ones you’d like to see first.

ACL object groups

I always thought that there was no need to restrict outbound sessions across a firewall in low-security environments. My last encounter with malware has taught me otherwise; sometimes we need to protect the rest of the Internet from our clumsiness. OK, so I decided to install an inbound access-list on the inside interface of my SOHO router that will block all SMTP traffic not sent to a well-known SMTP server (and let the ISP’s SMTP server deal with relay issues).

This is the point where my laziness kicked in: if I want to add another SMTP server in the future, I wouldn’t like to hack my ACL. I might also need to enter the SMTP server addresses in multiple ACLs and it would be annoying if I would add the server in one ACL but forget all the other related ACLs (because, you know, we don’t really need documentation). Fortunately, IOS release 12.4(20)T provides just the tool I need: the ACL object groups. I can define a group of host addresses and use them as an object in my ACL:

object-group network SMTP_Server
 description ISP SMTP server
ip access-list extended Inside
 permit tcp any object-group SMTP_Server eq smtp
 deny   tcp any any eq smtp log
 permit ip any any
interface Vlan1
 ip access-group Inside in

IOS implements network and service object groups. Network object groups can include hosts, IP prefixes or ranges. Service object groups define TCP, UDP or ICMP services (including all ACL options like ranges of ports). You can also nest object groups and define new groups as unions of already defined groups.

Control plane protection overview

Control plane (the main CPU that runs the routing protocols and all other application-layer services) is the most vulnerable part of your router. A determined attacker can quickly overload the CPU of any router (or switch) with a targeted denial-of-service attack, either by sending IP packets that are propagated from the switching fabric (or interrupt code on software-only platforms) to the control plane processes or by targeting individual services running on the router (see, for example, the problems one of the readers had with public DNS server running on the router).

Cisco IOS offers several control plane protection mechanisms. I’ve summarized them in the “Protecting the router’s control plane” article in the CT3 wiki and Sebastian Majewski has provided sample router configuration.

Becoming a spammer: hands-on experience

Reading the stories of Windows workstations becoming members of a spam botnet becomes way less enjoyable when you’re faced with the same problem (one of my kids managed to install a Trojan). It took me a day to clean the infected computer (it would have been easier to just format it, but the repeated installation of the Windows XP + Office software is so boring), but I’ve learned a few interesting networking lessons in the process that I’ll document in the next days.

Book review: Cisco Secure Firewall Services Module

I was very anxious to get my copy of Cisco Secure Firewall Services Module (FWSM) from Cisco Press, as I’m a purely router-focused person, and I wanted to understand the capabilities of the Firewall Services Module (PIX/ASA-like blade for the Catalyst 6500 switching system with virtual firewall capability). I have a good background in IOS-based firewalls and network address translation (NAT), so the book was a perfect fit for me. However, if you’re looking for “best practices for securing networks with FWSM,” you’ve been misled by the subtitle.

Off-topic: disappointed by the antivirus industry

One of my kids managed to get infected with a particularly sneaky Facebook Trojan: a link from a friend (probably also infected) pointed to a web page with a video that required installation of a newer version of the Flash player … which was actually the first part of the Trojan. It quickly downloaded a few more components and made itself cozy deep within Windows XP.

Before you start telling me that kids would click anything … we had “a few” not so very pleasant discussion after previous infections and they know not to open anything or click on something that looks strange. Unfortunately the update-happy industry has conditioned them to constant prompts to upgrade one or another component and the request to upgrade the Flash player was obviously too legitimate-looking.

Of course the workstations have anti-virus software which served me very well in the past. It identified the malware and claimed it had been quarantined. WRONG. Repeated scans with the same software always found the malware and claimed it has been cleaned. WRONG. On-line scanner from the same vendor identified a different malware and “removed” it. WRONG.

The worst part of the experience was a total lack of in-depth information that I became used to in the past (for example, the names of the infected files) as well as the claim that this is a “low threat” malware (which is why I was not alerted when the infection happened … if the anti-virus software tells you you’ve got low-threat infection and it has been cleaned, you don’t start panicking).

The only anti-virus package that really helped me was coming from an unbelievable source: Microsoft. Its monthly anti-malware program correctly identified four different Trojan components and pointed me to Microsoft’s anti-virus online solution, which contained all the information I needed, including the list of infected files that it could not remove. A safe-mode reboot, manual cleanup and a few more scans solved the problem.

After this experience, I’m left wondering. In the past, people claimed you should use anti-virus software from an independent source, and now it looks those sources are worse than Microsoft. Should I really give up and go for a one-vendor solution? Or should I reformat all the workstations in house and move to Fedora :). What are your experiences?

Interesting links | 2008-11-08

As always, Jeremy Stretch posted several interesting articles: how to hijack HSRP, introduction to split horizon in distance vector routing protocols and (long needed) default redistribution metrics.

Petr Lapukhov started playing with HTTP URL regular expressions within NBAR and documented his findings. The most interesting is the last Q/A pair: can I use NBAR as a content filtering engine?

And last but definitely not least, if you’re worried what will happen to WPA2 now that WPA has been cracked, Robert Graham explains the fundamental differences between WPA and WPA2. Also, make sure you read the detailed explanation of the WPA flaw to understand its implications.

Bidirectional Forwarding Detection

BFD is one of those simple ingenious ideas that make you wonder “Why did it take them so long to figure this out?” It’s a UDP-based protocol that replaces dozens of link-level failure-detection mechanisms and routing protocol tweaks with a simple, focused solution: detect hop-by-hop layer-3 failures.

I wanted to write about BFD a year ago when it was first advertised as being available in the low-end routers (BFD support on high-end platforms is much better, but I simply don’t have a GSR and a CRS-1 at home … yet), but it failed to work, so I had to shelve the idea until the IOS release 12.4(15)T matured to a point where BFD on ISR started working in IOS, not just in Powerpoint.

In this month’s IP corner article, “Improve the Convergence of Mission-Critical Networks with Bidirectional Forwarding Detection (BFD)”, I’m describing BFD principles, its configuration on Cisco IOS and give you practical examples how you can use BFD to improve next-hop failure detection.

Using Quagga in BGP tests

Quagga is a terrifically useful tool when you need to build a BGP test lab. Not only can you quickly add an extra BGP router in your network; it also allows you to insert BGP routes with almost any attribute you want. I’ve described some of its features and included a sample Quagga-to-router connectivity scenario in the “Use Quagga to generate BGP routes” article published in the CT3 wiki.

Isn’t Quagga extinct?

Those readers that have been discussing technical issues with me probably know that I rarely write something without testing it first. Somehow I didn’t feel like powering up our spare CRS, so you might wonder how I’ve tested the interoperability between four-byte AS implementations and Cisco IOS. Fortunately, there’s open-source routing protocol software suite named Quagga (which is an extinct subspecies of zebra in the real world) that has already implemented the new BGP standards and allowed me to do all the tests with just a router and a Linux host.

To help you get started, I wrote an article in the CT3 wiki describing the Quagga installation and configuration process on Fedora Linux.

Update@2008-11-07: Quagga is also available as binary package (RPM) for Red Hat/CentOS/Fedora, Solaris, Debian and Gentoo, but you'll most probably get at least a year old version. Vitaliy Gladkevitch provided RPM installation instructions.

OSPF Challenge #2: Final results

I’ve received almost a dozen responses to the second OSPF challenge, most of them correct. The key to the solution is the way OSPF checks neighbor’s IP address on point-to-point links (we already know that the subnet mask is ignored):

  • If the interface is unnumbered, the router ignores the source IP address in the OSPF hello packets.
  • If there’s an IP address configured on the interface, the router checks that the neighbor’s IP address (the source IP address in the OSPF hello packets) belongs to the same subnet. If the source IP address is not in the same subnet, the OSPF hello packet is ignored.

R1 and R2 (the router configuration can be found in the challenge) would establish adjacency only if the source IP address of the packets sent by R1 would be in the same subnet as the IP address on R2. Since the serial interface on R1 is unnumbered, R1 would use the IP address of the loopback interface in the OSPF hello packets. IP address of the loopback interface on R1 thus has to be in the subnet, giving you five choices (you cannot use the, and

However, as Yuri pointed out in his response, the routers do establish adjacency (so the challenge is solved) but do not build valid routing tables. The reason is the weird IP address used in the Link Data field of each unnumbered point-to-point link. According to RFC 2328, the router should use the MIB-II ifIndex as the IP address of an unnumbered interface. IOS performs subnet checks on SPF tree as well as on the OSPF hello packets and therefore R2 declares that R1 is not reachable. The following printout shows the R1’s router LSA as seen on R2:

R2#show ip ospf data router

            OSPF Router with ID ( (Process ID 1)

                Router Link States (Area 1)

  Adv Router is not-reachable
  LS age: 758
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID:
  Advertising Router:
  LS Seq Number: 80000016
  Checksum: 0x296B
  Length: 48
  Number of Links: 2

    Link connected to: a Stub Network
     (Link ID) Network/subnet number:
     (Link Data) Network Mask:
      Number of TOS metrics: 0
       TOS 0 Metrics: 1

    Link connected to: another Router (point-to-point)
     (Link ID) Neighboring Router ID:
     (Link Data) Router Interface address:
      Number of TOS metrics: 0
       TOS 0 Metrics: 64

All readers that sent me a correct response received a small award from our Remote Labs team: free access to the OSPF default mysteries e-lesson which includes a recorded presentation and three remote lab exercises.

Will we run out of BGP AS numbers?

In the “internet meltdown” post I’ve described the main reason for the routing problems we’re experiencing in the Internet: everyone wants to be truly multihomed. All these end-customers obviously need their own AS number and it’s no wonder the experts predict we’ll run out of AS numbers in two to three years.

There’s no need to panic: the technical solution (four byte AS numbers) has been ready for several years … but it’s not implemented yet in majority of Cisco IOS-based platforms. Does that mean we’ll experience Internet-wide problems when the regional registries start allocating AS numbers larger than 65536 in a few months? Luckily, the answer is NO, the new BGP standards are completely backward-compatible … but if you’re a Service Provider, you have to start thinking about the upgrade path.

You can find more answers on this topic
in the article I wrote for SearchTelecom.

The list of all articles I wrote for SearchTelecom is available in the CT3 wiki.

Was it really only a century ago?

This post brought back some ancient memories … and I’m always amazed how far we’ve got in the last 30 years. For me, it all started with an IBM 360, having 48K (forty eight kilobytes) of core memory in which it ran an operating system and three user partitions. Fortran IV was the only programming language and card reader the only input device.

Moving to a VAX 11/780 was a major improvement; it was a multitasking environment with real terminals. VAX was an interesting beast: the first step in the boot process was to start an embedded PDP-11 processor that read an 8” floppy disc and uploaded the microcode to the main CPU. The only drawback was that 30 users had to share 2M (two megabytes) of main memory and so I couldn’t crash the machine whenever I wanted.

A few years later, I managed to get access to a really cute research PDP-11 running RSX-11M. Finally I could start writing device drivers and kernel code without risking the wrath of dozens of users years older than myself. And then the personal computers appeared and I probably made one the best choices I could – the BBC Micro from Acorn. It was never popular, but it had an amazingly well-designed operating system that you could extend in any way you wish (and even symbolic assembly language built into its BBC BASIC).

Control and Data planes in a router

Every now and then I feel the need to write a very basic article, explaining the foundations and getting the terminology straight. Today I’m trying to explain the control and data planes in a router (or a layer-3 switch, depending on your marketing bias). Your opinions, fixes, corrections and all other comments are most welcome.

OSPF challenge #2: mixing numbered and unnumbered interfaces

Update: This challenge is closed, see the final results (November 4th 2008).

Assuming you have the following configurations on R1 and R2:

hostname R1
interface Loopback 0
 ip address
interface Serial 0
 encapsulation ppp
 ip unnumbered Loopback0
 ip ospf 1 area 1
router ospf 1
hostname R2
interface Serial 0
 encapsulation ppp
 ip address
 ip ospf 1 area 1
router ospf 1

What IP address can you use on the loopback interface of R1 to establish adjacency between R1 and R2? Can you use more than one IP address?

This challenge was triggered by a comment uri wrote on the “OSPF ignores subnet mask mismatch on point-to-point links” post, claiming that you cannot mix numbered and unnumbered interfaces in OSPF.

OSPF Challenge #1: Final results

I’ve received several e-mails responding to the mismatched OSPF subnet challenge. Some of the readers claimed that the configuration would work as-is; if you were one of them, I would advise you do some lab test the next time. A few of the respondents also noted that it was more a review question than a challenge (since I’ve been writing about this topic a few days back) and everyone who decided the configuration has to be fixed has provided the correct solution: you have to configure the Fast Ethernet as a point-to-point OSPF interface and the routers stop complaining about the OSPF subnet mask mismatch.

Unfortunately, someone decided to prevent everyone else from having real fun figuring out the solution and posted the solution as a comment to my post almost immediately after I wrote it (but I’m positive that those readers that sent me e-mails did not read that comment first). Lesson learned: the next time I’ll disable comments in the challenges.

Our remote labs team provided small awards to all readers that have sent me an e-mail with a correct solution: free access to the "When OSPF becomes a distant vector protocol" e-lesson.

Who is an associate?

One of my readers made an interesting observation in response to my “Knowledge or recipes” post: maybe network associates need recipes more than knowledge. My first reaction was to disagree; in my understanding, technicians work with recipes, engineers need to know what they’re doing and why. But then I tried to figure out what the term “networking associate” really means.

New in NIL Community | 2008-10-27

Jeremy Stretch has been extremely active in the CT3 wiki in last few days, writing about OSPF inter-area routing and various aspects of PIM. His articles cover the basics of PIM, principles of PIM dense mode, more detailed overview of PIM sparse mode and a hint what Bidirectional PIM is.

We've been discussing basic OSPF operations in the NIL forum. If you're aware of any good online resource explaining OSPF basics, SPF algorithm, the resulting SPF tree and the OSPF cost calculation, I would appreciate if you could post the link(s); preferably in the forum or as a comment to this post. If it turns out there's nothing really useful (which I doubt) I could write something, but I would rather spend time on more advanced (but probably less popular :) topics.

Embedded comment form

I've enabled the new "embedded comment form" in Blogger. This should make writing the comments a bit easier ... but Blogger is known to report weird errors every now and then when you're using the embedded form. If that happens to you, just press the "Publish" button yet again (after entering another CAPTCHA if needed). I hope you'll find that the ease-of-use of the new form outweighs the occasional hiccups (and I won't even try to open a case with Google about this).

Telnet access restrictions

A while ago I've got an interesting question from one of the readers:

I'd like to be able to configure a set of routers to only be manageable from each other. Something like an access-class matching minimum packet TTL would probably be good enough, better if some connected routes could be tagged and access granted based on that. The idea is to keep router-by-router logins in case of routing problems, without opening up access too widely.

I did a few tests with IOS release 12.4(15)T and neither access-class nor control-plane policing recognizes the TTL field in ACL (various bits and pieces of IOS use the same data structures in different procedures, thus resulting in inconsistent behavior). Alternatively, you could deploy inbound access lists on all interfaces, but this is probably way too cumbersome to manage.

The best you can do without going into weird solutions is to allocate router loopback interfaces and inter-router links from a tightly controlled address space and only allow telnet from that address space (while at the same time filtering IP packets pretending to come from that same address space on the perimeter of your network). As the IOS supports extended access lists in the access-class line configuration command, you could allow SSH from a wider set of IP addresses and limit Telnet to the address range allocated to inter-router links.

IP address lookup

Someone recently asked me how to get the physical location of an IP address. One of the better (free) services available on the Internet is the IP2Location (demo) service.

This feature might come handy if you're trying to figure out who's attacking your application servers (when the TCP session has already been established). Denial-of-service attacks commonly use fake source IP addresses.

OSPF area configuration best practices

The assignment of router’s interfaces into OSPF areas should be a non-issue these days, but for whatever reason some of the students I’m mentoring still use the ridiculous practice that was promoted in older learning materials: a separate network statement using IP subnet and inverse subnet mask for every single interface. I’ve documented what I consider to be best practices in the “OSPF area configuration best practices” article in the CT3 wiki. If you disagree with my opinion, please feel free to edit my article or share your thoughts in a comment to this post.

Read more in the CT3 wiki

Gaining Knowledge - what’s the best way to do it?

A few days after my “Knowledge or Recipes” post, Greg Ferro started his “Experience or Certifications” series with a radical “I would always choose certification over experience” approach that quickly moderated into “Knowledge is more fundamental than experience … but you need both”. It’s nice to see someone else thinking along the same lines as yourself.

OSPF challenge #1: Establish OSPF adjacency on a LAN interface

Update: This challenge is closed, see the final results (October 29th 2008).

You could get something like this only in a CCIE lab (I would hope): R1 and R2 should establish OSPF adjacency, but you cannot change or remove any of the existing configuration commands (you can add new commands).

hostname R1
interface FastEthernet 0/0
 ip address →
 ip ospf 1 area 1
router ospf 1
hostname R1
interface FastEthernet 0/0
 ip address →
 ip ospf 1 area 1
router ospf 1

OSPF breaks when faced with overlapping IP addresses

A while ago cciepursuit described his problems with PPP-over-Frame Relay. Very probably his problems were caused by a static IP address assigned to the virtual template interface (this address gets cloned to all virtual access interfaces and IOS allows you to have the same IP address on multiple WAN point-to-point links). I recreated very similar (obviously seriously broken) scenario in my lab using point-to-point subinterfaces over Frame Relay to simplify the setup.

This is not something you’d want to do in your production network.

The relevant parts of router configurations are included below:

S1#show run | section interface Serial
interface Serial1/0
 no ip address
 encapsulation frame-relay
interface Serial1/0.100 point-to-point
 description Link to C1
 ip address
 frame-relay interface-dlci 100

S2#show run | section interface Serial
interface Serial1/0
 no ip address
 encapsulation frame-relay
 frame-relay lmi-type ansi
interface Serial1/0.100 point-to-point
 description Link to C1
 ip address
 frame-relay interface-dlci 100

C1#show run | section interface Serial
interface Serial1/0
 no ip address
 encapsulation frame-relay
interface Serial1/0.100 point-to-point
 description Link to S1
 ip address
 frame-relay interface-dlci 100
interface Serial1/0.101 point-to-point
 description Link to S2
 ip address
 frame-relay interface-dlci 101

I expected the distance vector protocols to work flawlessly, as they track the next-hop as well as inbound interface over which the routing update was received. With the following RIP configuration on all three routers …

router rip
 version 2
 no auto-summary

… the routing table on C1 looked almost OK, apart from the weird entry for the subnet:

C1#show ip route | begin Gateway
Gateway of last resort is not set is variably subnetted, 5 subnets, 3 masks
C is directly connected, Serial1/0.100
                    is directly connected, Serial1/0.101
R [120/1] via, 00:00:01, Serial1/0.101
C is directly connected, Loopback0
R [120/1] via, 00:00:02, Serial1/0.100

The next routing protocol was EIGRP; configuration was very similar to the RIP case:

router eigrp 1
 no auto-summary

Yet again, the IP routing table looks as expected:

C1#show ip route | begin Gateway
Gateway of last resort is not set is variably subnetted, 4 subnets, 2 masks
C is directly connected, Serial1/0.100
                    is directly connected, Serial1/0.101
D [90/2297856] via, 00:02:33, Serial1/0.101
C is directly connected, Loopback0
D [90/2297856] via, 00:02:33, Serial1/0.100

OSPF also behaved as expected, producing weird results. I never got both remote routes in the IP routing table on C1; one of them (or even both of them) was missing. This is not surprising; OSPF builds the SPF tree from the topology database and gets totally confused when two interfaces have the same IP address.

The information in the topology database is correct; in my lab, C1 advertised that it can reach S1 and S2 (both of them appeared as router-to-router links in the router LSA) …

C1#show ip ospf database router self-originate

            OSPF Router with ID ( (Process ID 2)

                Router Link States (Area 1)

  LS age: 283
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID:
  Advertising Router:
  Number of Links: 5

    Link connected to: a Stub Network
     (Link ID) Network/subnet number:
     (Link Data) Network Mask:
      Number of TOS metrics: 0
       TOS 0 Metrics: 1

    Link connected to: another Router (point-to-point)
     (Link ID) Neighboring Router ID:
     (Link Data) Router Interface address:
      Number of TOS metrics: 0
       TOS 0 Metrics: 64

    Link connected to: a Stub Network
     (Link ID) Network/subnet number:
     (Link Data) Network Mask:
      Number of TOS metrics: 0
       TOS 0 Metrics: 64

    Link connected to: another Router (point-to-point)
     (Link ID) Neighboring Router ID:
     (Link Data) Router Interface address:
      Number of TOS metrics: 0
       TOS 0 Metrics: 64

    Link connected to: a Stub Network
     (Link ID) Network/subnet number:
     (Link Data) Network Mask:
      Number of TOS metrics: 0
       TOS 0 Metrics: 64

… but OSPF got totally confused when trying to build the SPF tree, deciding that one (or both) of remote routers was unreachable:

C1#show ip ospf database router adv-router

            OSPF Router with ID ( (Process ID 2)

                Router Link States (Area 1)

  Adv Router is not-reachable
  LS age: 356
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID:
  Advertising Router:
  Number of Links: 3

    Link connected to: another Router (point-to-point)
     (Link ID) Neighboring Router ID:
     (Link Data) Router Interface address:
      Number of TOS metrics: 0
       TOS 0 Metrics: 64

    Link connected to: a Stub Network
     (Link ID) Network/subnet number:
     (Link Data) Network Mask:
      Number of TOS metrics: 0
       TOS 0 Metrics: 64

    Link connected to: a Stub Network
     (Link ID) Network/subnet number:
     (Link Data) Network Mask:
      Number of TOS metrics: 0
       TOS 0 Metrics: 1

In my humble opinion, IOS could do better (the topology table has enough information to build the correct SPF tree), but this is a nonetheless a broken design that a router should never be exposed to.

Summary: Virtual template interfaces should be unnumbered to prevent address overlap on virtual access interfaces. If you insist on using a fixed IP address on virtual template interfaces, don’t expect OSPF to work.

OSPF ignores subnet mask mismatch on point-to-point links

The common wisdom says that the subnet mask mismatch will stop the OSPF adjacency from forming (I’ve included a sample debugging printout in yesterday’s post). In reality, the subnet mask is checked only on the multi-access interfaces and is ignored on point-to-point links. The source of this seemingly weird behavior is Section 10.5 of RFC 2328 which says:

The generic input processing of OSPF packets will have checked the validity of the IP header and the OSPF packet header. Next, the values of the Network Mask, HelloInterval, and RouterDeadInterval fields in the received Hello packet must be checked against the values configured for the receiving interface. Any mismatch causes processing to stop and the packet to be dropped. In other words, the above fields are really describing the attached network's configuration. However, there is one exception to the above rule: on point-to-point networks and on virtual links, the Network Mask in the received Hello Packet should be ignored.

Cisco conforms strictly to the RFC and allows OSPF neighbors to form adjacency over a point-to-point link (Frame Relay subinterface in my test lab) even when the subnet masks don't match. The routers in the lab happily formed the OSPF adjacency even though I've used a /24 mask on one end of the link and a /30 mask on the other end:

S1#show ip ospf interface brief
Interface    PID   Area    IP Address/Mask    Cost  State Nbrs F/C
Se1/0.100    1     1        64    P2P   1/1
Lo0          1     1        1     LOOP  0/0
S1#show ip ospf neighbor Serial1/0.100
Neighbor ID     Pri   State       Dead Time   Address    Interface         0   FULL/  -    00:00:38   Serial1/0.100

C1#show ip ospf interface brief
Interface    PID   Area    IP Address/Mask    Cost  State Nbrs F/C
Fa0/0        1     0        10    BDR   1/1
Lo0          1     0       1     LOOP  0/0
Se1/0.100    1     1        64    P2P   1/1
Se1/0.101    1     1          64    P2P   1/1
C1#show ip ospf neighbor Serial1/0.100
Neighbor ID     Pri   State           Dead Time   Address         Interface          0   FULL/  -        00:00:37        Serial1/0.100

This behavior was brought to my attention by Shahid Rox. Thanks.

Troubleshooting OSPF adjacencies

Troubleshooting OSPF adjacencies can be a nightmare: if you’ve misconfigured the OSPF interface parameters (the timers or the subnet mask), the adjacency will not form, but the router will not tell you why. The only mechanism you can use to detect the mismatch is the debug ip ospf hello command … just don’t try to use it on a console session of a router running OSPF across hundreds of interfaces.

The OSPF hello event debugging does not display OSPF packets received from a different subnet. If you configure mismatched IP subnets (not the subnet mask) on adjacent routers, you will not see any received hello packets.

To limit the debugging outputs to a single interface, use the debug interface command.

For example, in my test network, the routers did not want to establish adjacency on the Fast Ethernet interface:

C1#show ip ospf interface brief
Interface    PID   Area    IP Address/Mask    Cost  State Nbrs F/C
Fa0/0        1     0        10    DR    0/0
Lo0          1     0       1     LOOP  0/0
Se1/0.101    1     1          64    P2P   1/1
Se1/0.100    1     1          64    P2P   1/1
C1#show ip ospf neighbor

Neighbor ID  Pri   State        Dead Time   Address    Interface       0   FULL/  -     00:00:38   Serial1/0.101       0   FULL/  -     00:00:36   Serial1/0.100

Using the OSPF hello debugging limited to Fast Ethernet interface I quickly discovered the source of the problem: the subnet mask mismatch between the adjacent routers.

C1#debug interface FastEthernet 0/0
Condition 1 set
C1#debug ip ospf hello
OSPF hello events debugging is on
OSPF: Rcv hello from area 0 from FastEthernet0/0
OSPF: Mismatched hello parameters from
OSPF: Dead R 40 C 40, Hello R 10 C 10  Mask R C

Quality in training: you can make a difference

Several comments I’ve received in response to my “Knowledge or recipes” post were slightly resigned, leading me to the unfortunate conclusion that you all gave up and decided to live with the current state of the IT training business. But you can do something about it – go out and vote!

Display interfaces belonging to a single OSPF process

I’m constantly receiving interesting OSPF-related queries. Obviously the many hidden details of the OSPF specs result in slightly unexpected behavior and constant amazement of engineers studying OSPF. During this week, I’ll focus on a few interesting OSPF intricacies.

Let’s start with an easy one: I’ve already described how you can use the show ip ospf interface brief command if you want to display the OSPF interface status (including the interface area, OSPF cost, link type and router status on broadcast links). Unfortunately, this command does not allow you to specify the OSPF process ID and displays interfaces belonging to all OSPF processes (if you run multiple OSPF processes on the router).

Here is a sample printout taken from a router running OSPF processes #2 and #13:

C1#show ip ospf interface brief
Interface    PID   Area    IP Address/Mask    Cost  State Nbrs F/C
Lo102        2     22        1     LOOP  0/0
Fa0/0        13    0        10    BDR   1/1
Lo0          13    0       1     LOOP  0/0
Se1/0.101    13    1          64    P2P   1/1
Se1/0.100    13    1          64    P2P   1/1

You can use an output filter to display the interfaces belonging to a single OSPF process. The filter is quite convoluted …

C1#show ip ospf interface brief | include ^[^ ]+ +13
Fa0/0        13    0        10    BDR   1/1
Lo0          13    0       1     LOOP  0/0
Se1/0.101    13    1          64    P2P   1/1
Se1/0.100    13    1          64    P2P   1/1 

… and works like this:

  • The initial caret (^) matches the beginning of the line, ensuring that our filter will match exactly what it needs to match. Without the initial caret, the filter could generate a match anywhere in the line, potentially resulting in false positives.
  • The [^ ]+ pattern matches any non-empty (the + sign) string of non-space characters (the [^ ] expression matches anything but the whitespace). This part of the pattern matches the interface name.
  • The  + pattern matches the string of spaces between the interface name and the process ID.
  • The final part of the pattern (13) matches the OSPF process ID.

You can transform this complex output filter into an easy Tcl script. See the “Simple extensions to exec-mode CLI” and “Simple CLI extensions: handling special characters” posts.

Interesting links | 2008-10-12

MPLS on 7600: the devil is in the details

I've got a simple question recently: “Can I run MPLS on a VLAN interface on 7600?” My initial response was “Sure, why not.”, as I knew we've deployed MPLS in 7600-based networks and there should be no significant difference between a routed port and a VLAN interface on a 7600 (this box treats everything as a VLAN internally).

It turned out the problem was "a small detail" that's not advertised in any 7600-related MPLS marketing material on Cisco web site: you need Advanced IP Services software to run MPLS. To make matters worse, the only mention of 7600-series devices in the Cisco IOS Packaging Product Bulletin I've finally found within the 7600 routers product literature is in the first marketing slide.

This article is part of You've asked for it series.