Your browser failed to load CSS style sheets. Your browser or web proxy might not support elliptic-curve TLS

Building network automation solutions

6 week online course

reserve a seat

Basic MPLS VPN design & configuration guidelines

I’ve collected very basic MPLS VPN design and configuration guidelines and squeezed them into a two page PDF document. If you’re new to MPLS VPN, you’ll probably find it useful … although reading the MPLS and VPN Architectures book still remains the mandatory next step.

Download the document from the CT3 wiki

see 11 comments

Is it wise to use certifications in the candidate selection process?

My previous certification-related post described how some companies use certifications to filter job applicants for networking-related positions. Should you follow that example? If you’re in a country with a saturated job market, where the number of applicants far exceeds the number of job postings (consider yourself very lucky if you’re an employer), you should certainly use whatever filters you can to screen the hundreds of applications you receive … but be aware that you have potentially lost a few gems hidden in the flood.

read more Add comment

MPLS support on 1800-series routers

Christoph sent me an interesting question a few days ago:

I played a bit arround with 2 Cisco 1803 and I found MPLS related configurations commands in IOS 12.4(15)T (Advanced Enterprise) on this box. MPLS was not listed as a included fearture in the Cisco Feature Navigator for this image and some searching at took me to a 2 year old document telling me that MPLS isn't supported on this series. Some more searching took me back to the Cisco Feature Navigator which lists MPLS as feature for the Cisco 1805 router (which uses the same IOS image, afaik).
So, I'm a bit confused now if MPLS is really working / supported on the low-end Cisco ISR 1800 fixed series?

MPLS was mostly available but never supported on low-end platforms (including Cisco 2600). In those days I've taken some heat for reusing existing 2600-based labs to teach Cisco-internal MPLS courses (since we were teaching the students to configure unsupported devices :).

Anyhow, the "not supported" means exactly that: it may be available (well, it is), it may work (it actually does), but if it's broken (and I've seen at least one low-end-platform-specific bug in the early days) you can't complain.

Is anyone aware whether the official support for the MPLS on 1800 series has changed? If so, please share your information with us.

If you need to offer a production-grade service to your customers, don't use unsupported equipment; if you need a solution for your personal needs or you're building a lab, go ahead.

see 10 comments

Why would I need a Web Application Firewall?

If you have been visited by a friendly Cisco sales engineer recently, you might have already heard about the ACE Web Application Firewall (WAF). If you’re curious enough to start investigating on your own, you might have stumbled across the WAF product description on Cisco’s Web site, which tells you … nothing.

Let’s start with an easy question: if I already have a firewall, why would I need another box with “fire” and “wall” in its name? The short answer is “Because Web programmers rarely know how to write secure Web applications.”

read more Add comment

Correction: NAT-translated DNS responses are not cacheable

It looks like the wording in the “NAT-translated DNS responses are not cacheable” post was a bit too vague, as some readers understood the router would mess the TTL field in the DNS response payload when changing the IP addresses in the IP header of the response packet.

That's not the case; the TTL field in the DNS response payload is touched only if the router performs application-layer translation of the DNS response (for example, changing the A record in the DNS response). I've reworded the original post; I can only hope I've made it unambiguous (after all, English is not my native language).
Add comment

NAT-translated DNS responses are not cacheable

Slightly reworded on 2008-12-18. Not only is NAT (as implemented in Cisco IOS) very picky about the translation of IP addresses in the payload of DNS responses (it translates only addresses defined in IP-level NAT translations with no additional route-map filters), it also sets the TTL field in the DNS response to zero when translating an IP address in the DNS response payload based on an existing L3 NAT entry making the DNS response completely uncacheable.

The behavior makes some sense, as the L3 NAT entry might change before the DNS response expires, but the implementation is definitely overly aggressive. In my opinion, IOS should use some sensible (configurable?) value for static NAT translations and times comparable to NAT timeouts for dynamic NAT translations. What do you think?

see 4 comments

Book review: Voice over IP Security

Based on the title, I would assume that the Cisco Press book Voice over IP Security: Security best practices derived from deep analysis of the latest VoIP network threats attracts primarily senior voice engineers who know that they have to secure their production networks. The author of the book strongly disagrees with my opinion, however, spending more than a third of the book on baseline explanations of VoIP, SIP, H.323, firewalls, NAT, DES, IPSec…. I enjoyed the overview chapters, as I last configured VoIP before SIP was invented, but an experienced VoIP engineer would be disappointed.

read more Add comment

This is QoS; who cares about real-time response?

It all started with a innocuous question: can you detect voice traffic with EEM? Looks simple enough: create a QoS class-map that matches voice calls and read the cbQosClassMapStats table in the CISCO-CLASS-BASED-QOS-MIB. The first obstacle was finding the correct indexes, but a Tcl script quickly solved that; I was ready to create the EEM applet. The applet failed to work correctly and after lots of debugging I figured out the counters in the cbQosClassMapStats table change only every 10 seconds.

I couldn’t believe my eyes and simply had to test other MIB variables as well. As expected, the IF-MIB (standard interface MIB) counters increase in real-time, but obviously someone had the bright idea that we need to detect changes in traffic profile only every now and then. Although I've received suggestions from my readers, none of them works on an 1800 or a 7200. Oh, well, Cisco developers from the days when I started working with routers would know better.

To test the MIB variable behavior I wrote a simple Tcl script to test the MIB variables. It reads the specified MIB variable at fixed intervals and prints the values, so you can monitor the changes in the MIB variable in real-time. I started low-bandwidth UDP flood across the router and monitored the output bytes interface counter. As expected the counter changed in real time and accurately tracked the amount of traffic sent through the router.

GW#pm ifOutOctets.3 public 10 1000          
polling ifOutOctets.3 for 10 seconds (10 iterations)

 0.000 ifOutOctets.3=42528679        
 1.000 ifOutOctets.3=42537767        
 2.000 ifOutOctets.3=42546713        
 3.000 ifOutOctets.3=42555719        
 4.000 ifOutOctets.3=42564665        
 5.000 ifOutOctets.3=42573611        
 6.000 ifOutOctets.3=42582699        
 7.000 ifOutOctets.3=42591645        
 8.000 ifOutOctets.3=42600591        
 9.000 ifOutOctets.3=42609537        

Then I created a simple class-map and policy map

GW#show policy-map interface

  Service-policy output: LAN

    Class-map: Voice (match-all)
      20438 packets, 2902196 bytes
      30 second offered rate 70000 bps
      Match: access-group name Voice

    Class-map: class-default (match-any)
      41 packets, 3967 bytes
      30 second offered rate 0 bps, drop rate 0 bps
      Match: any 

… and monitored the pre-policy byte counter (cbQosCMPrePolicyByte64) for the Voice class. The value changed only once every ten seconds:

GW#pm cbQosCMPrePolicyByte64.50.10767521 public 10 1000
polling cbQosCMPrePolicyByte64.50.10767521 for 10 seconds (10 iterations)

 0.000 cbQosCMPrePolicyByte64.50.10767521=0x002865366
 1.000 cbQosCMPrePolicyByte64.50.10767521=0x002865366
 2.000 cbQosCMPrePolicyByte64.50.10767521=0x002865366
 3.000 cbQosCMPrePolicyByte64.50.10767521=0x002865366
 4.000 cbQosCMPrePolicyByte64.50.10767521=0x002865366
 5.000 cbQosCMPrePolicyByte64.50.10767521=0x002865366
 6.000 cbQosCMPrePolicyByte64.50.10767521=0x00287acf8
 7.000 cbQosCMPrePolicyByte64.50.10767521=0x00287acf8
 8.000 cbQosCMPrePolicyByte64.50.10767521=0x00287acf8
 9.000 cbQosCMPrePolicyByte64.50.10767521=0x00287acf8

What can I say ... apart from expressing my deepest disappointment :(

see 14 comments

The most convoluted MIB I’ve seen

Jared Valentine sent me a really interesting problem: he would like to detect voice traffic and start shaping TCP traffic for the duration of the voice call. The ideal solution would be an EEM applet reacting to the changes in the CISCO-CLASS-BASED-QOS-MIB; one of its tables contains the amount of traffic for each class configured in a service policy.

The MIB navigation looks simple: you just read the values from the cbQosClassMapStats table, indexed by policy ID and class ID. The real problem is finding the correct index values. I could walk the MIB manually with a MIB browser or snmp_getnext TCL calls, but this approach is obviously not scalable, so I wrote a script that walks through the cbQosServicePolicy, cbQosObjects, cbQosPolicyMapCfg and cbQosClassMapCfg tables and prints the index values you need.

You can find more Tcl-related information in the Tclsh on Cisco IOS tutorial. Sample Tclsh scripts are available in the Tclsh script library. If you need expert help in planning, developing or deploying Tclsh scripts in your network, contact the author.
see 3 comments

Cisco AXP Developer Contest

Do you have a great idea what a router should be able to do, but you cannot make it work because you can't tweak router's embedded management tools (including Tcl)? Here is an opportunity you shouldn't miss: enroll in the Cisco AXP Developer Contest. Even if you're absolutely positive that having an extra Linux server sitting next to the router is better than a router blade (and some Linux pundits had lots to say on the topic), sending your ideas to Cisco won't cost you more than a few moments of your time and an e-mail ;)

And, BTW, mundane (but nonetheless highly useful) ideas like »visual policy map editor« or truly integrated DNS/DHCP server probably won't get you very far (that's why I'm not applying :).

Full disclosure: I was kindly asked if I could help spreading the word about the contest, but all the enthusiasm is exclusively mine.

see 3 comments

Certifications: a new barrier to entry

Recent blog posts indicate that, in at least some market segments, IT certifications are becoming a new barrier to entry: companies require a specific set of certifications in their job offerings and use those requirements to filter the candidates who are invited to the initial interview. Obviously, IT vendors pushing the certifications are getting some real traction. On the other hand, anecdotal evidence indicates that certification holders are sometimes able to memorize vast amounts of information without being able to put it to use (I don’t want to imply that they used other, less honest methods).

read more Add comment

NAT translation logging

The ip nat log translations syslog command starts NAT logging: every NAT translation created on the router is logged in syslog (which can be sent to console, syslog host or internal buffer). You could use this command as a poor man’s reporting tool if you have to monitor the address translations on your edge routers (for example, due to security policy or auditing requirements). Obviously you should configure the no logging console first in a production environment; otherwise your router will hang a few moments after you’ve enabled NAT logging.

The logging printouts include:

  • layer-4 protocol;
  • inside local and global addresses and port numbers;
  • outside local and global addresses and port numbers.

For example, the following messages were logged when an inside host tried to access web server at the IP address The inside source address was translated through a NAT pool (configured with ip nat inside source list ACL pool POOL command) into and the inside destination address was translated into outside destination address (configured with ip nat outside source static command).

12:17:12.503: %IPNAT-6-NAT_CREATED: Created tcp
12:18:47.751: %IPNAT-6-NAT_DELETED: Deleted tcp

Does it sound weird to specify the destination address translation with the ip nat outside source configuration command? It does to me …

see 7 comments

Online session poll results

I would like to thank everyone that took time and voted on the subject of my next on-line session. We have a clear winner: Dynamic routing across a firewall … although the closeness of the other topics might indicate a helping hand of our polish friend. Unfortunately, I have some bad news to go with the good ones – the “December” session will most likely happen in January or February (I'm simply running out of time).
Add comment

Update interval for IOS MIB counters?

Dear readers! This time I really need your help (uncle Google and his relatives gave me only one relevant hit and even that doesn't work on 7200 or 1800).

I'm trying to implement an EEM applet that would detect traffic rate change using CISCO-CLASS-BASED-QOS-MIB. Everything would work perfectly ... if only IOS wouldn't update the MIB counters approximately every 10 seconds, not in real-time. Is anyone aware of a configuration command that would force the router to update these counters any faster?
see 11 comments

Random “Scenic Route Certification” thoughts

The “Sometimes the path is more important than the destination” post has generated numerous highly interesting comments. I already planned to write about some of the issues raised by the readers (certification grind mill) or wrote about others (knowledge or recipes), so I’ll skip those and focus on the other interesting bits-and-pieces (but please make sure you read the original post first).

read more see 3 comments

Fix the EEM RFC 822-related bug

A while ago I’ve reported that Cisco IOS does not conform to RFC 822, potentially resulting in ill-formed e-mails send from EEM applets. The bug has been fixed in IOS releases 12.4(15)T2 (and all later 12.4T releases) and 12.2(33)SRC, but if you’re stuck with an older release, you can use a nice trick Gerald Kowalsky sent me.

Read more in CT3 wiki

Add comment

Should VTP be disabled by default?

One of my readers sent me a question that triggered one of my old grudges:

In my experience, when you first add a new switch (having a NULL domain) on an existing VTP Domain, it inherits the domain name, regardless of it being a VTP Server. I was wondering if this is a feature (i.e. has proved to be a solution in most cases) or a bug (i.e. has proved to cause problems in most cases). I know it's proved to be the latter for us!

In my personal opinion Cisco at one point in time wanted too much plug-and-play and someone had a great idea that you can just plug another switch into your network and it would autoconfigure itself. We've been suffering because of that "insight" ever since (and the CCIE written test has material for a few more interesting questions :).

I strongly believe that VTP should be turned off by default and should generate a warning before being enabled, but it will probably not happen. What do you think?

Disclaimer: I am not a switching person and have no idea about anything below or above layer 3.

see 16 comments

Configuring IS-IS on partially-meshed multi-access WAN network

If someone insists on running IS-IS on a partially-meshed Frame Relay network without using subinterfaces (which is the proper way of doing it), it can still be done … assuming you understand what’s going on and properly configure the IS-IS routers.

Read the configuration guidelines in the CT3 wiki.

Add comment

I’m too old … I prefer CLI over GUI

I was delighted when I got access to Cisco’s Application Control Engine (ACE) XML Gateway/Web Application Firewall (WAF) box. This box is the perfect intersection of three fields that really interest me: networking, security and Web programming. To my huge disappointment, though, all the real configuration can only be done through the Web interface. I understand that casual users of a device prefer a graphical user interface (GUI) over text commands (and Generation Z has never seen a terminal window, DOS prompt or, God forbid, an actual terminal), but you can achieve so much more with a simple text-based configuration approach:

read more see 10 comments

ACE XML Gateway Poll Results

Thanks to everyone who took time to participate in my “ACE XML Gateway” poll (and special thanks to our anonymous friend from Poland who amused numerous readers and helped me set up slightly more secure polls). A large majority (with the notable and vocal exception of the previously-mentioned anonymous contributor) would like to see posts (or at least pointers to them) in the IOS Hints blog:
see 2 comments

When would an MPLS LSR have Untagged output label?

This is a nice MPLS question I’ve received from one of the readers:

I have understood the Penultimate Hop Popping (PHP) process, but I don’t understand when a router would use UNTAGGED instead of POP TAG?

Instead of answering the question directly, let's walk through a series of simple Q&A pairs that will help you understand the whole process (remember: knowledge, not recipes!).

It's highly recommended you read the first few chapters of the MPLS and VPN Architectures book before the rest of this post.

Where does the Untagged keyword appear? It only appears as the output label in the LFIB (Label Forwarding Information Base) that you can inspect with the show mpls forwarding-table.

What does the Untagged keyword mean? This keyword means that the router has no output label associated with the forwarding equivalence class (FEC ... usually an IP prefix). Since there is no output label, the router cannot perform a label swap (or pop) but has to remove the whole MPLS shim header.

Where would a router get the output label? It's received from the next-hop router.

When would a router have no output label? When there is no next-hop router or when the next-hop router did not advertise a label for the IP prefix.

When would there be no next-hop router? If the IP prefix is a directly connected subnet (including a loopback interface) or a summary route advertised by the router itself.

When would the next-hop router not advertise a label? The reasons a next-hop router would not advertise a label for an IP prefix include:

  • It's not running MPLS.
  • It's running MPLS but not CEF (MPLS labels are assigned to IP prefixes in CEF table).
  • It's not reachable across an MPLS-enabled interface (both routers could be running MPLS, but the transit interface does not have the mpls ip configuration).
  • The LDP session has not been established yet.
  • There is a mismatch in LDP protocol (one router is running Cisco's proprietary TDP, the other one standard LDP).
  • The next-hop router uses an access-list to filter the IP prefixes for which the MPLS labels are advertised.

Summary: you would see the Untagged label in the LFIB when the IP prefix is a directly connected interface, a summary route or the next-hop router has not advertised the label.

see 6 comments