I need to slow down :)
I’ve just opened the January Technical Services News from Cisco. Nothing in there that would really interest me. Almost no routing protocols (one OSPF article), no BGP, no MPLS VPN. Based solely on this newsletter, one could get the feeling that I’m producing more documents covering core IP routing in a month than Cisco (I am positive that’s not the case).
But maybe Cisco’s engineers are refocusing on the new Support Wiki. Not really. After I’ve filtered out sequential changes to a single document, there were only 11 significantly changed documents in the Support Wiki in the last 30 days.
So I’m left wondering … what’s going on? Has everything already been written about the core IP routing features and the productive minds have shifted to voice and wireless? Are the engineers focused on IP routing becoming the dinosaurs? What’s your perspective?
But one thing is clear: I need to slow down.
Interactions between IP routing and QoS
One of my readers sent me an interesting question a while ago:
I reviewed one of your blog posts "Per-Destination or Per Packet CEF Load Sharing?" and wondered if you had investigated previously on how MQC QoS worked together with the CEF load-sharing algorithm (or does it interact at all)? For example, let's say I have two equal cost paths between two routers and the routing table (as well as CEF) sees both links as equal paths to the networks behind each router. On each link I have the same outbound service policy applied with a simple LLQ, BW, and a class-default queues. Does CEF check each IP flow and make sure both link's LLQ and BW queues are evenly used?
Unfortunately, packet forwarding and QoS are completely uncoupled in Cisco IOS. CEF performs its load balancing algorithm purely on source/destination information and does not take in account the actual utilization of outbound interfaces. If you have bad luck, most of the traffic ends on one of the links and the packets that would easily fit on the other link will be dropped by the QoS mechanisms.
You could use multilink PPP to solve the problem in low-speed environments. With MLPPP, CEF sends the traffic to a single output interface (the Multilink interface) and the queuing mechanisms evenly distribute packet fragments across the links in the bundle.
In high-speed environments, you can only hope that the number of traffic flows traversing the links will be so high that you’ll get a good statistical distribution (which is usually the case).
Flash-based DHCP database
Pete sent me an interesting question a while ago:
It might be interesting to write an article about ip dhcp database flash:dhcp-db command, documenting the pros of surviving a reboot versus cons of wear on the flash device.
I’ve already written about a few problems that can be solved with the DHCP database (but obviously a longer text is warranted … already stored in my to-do list) and it took me a while to find the time to dig out the relevant information on the flash device wear.
EBGP Multipath Load Sharing and CEF
When I was discussing the details of the BGP troubleshooting video with one of my readers, he pointed out that I should mention the need for CEF switching in EBGP multipath scenario. My initial response was “Why would you need CEF? EBGP multipath is older than CEF” and his answer told me I should turn on my gray cells before responding to emails: “Your video as well as Cisco’s web site recommends CEF for EBGP multipath design… but interestingly, it does work without CEF”.
The real reason we need CEF in EBGP load sharing designs is the efficacy of load distribution. Without CEF, the router will send all traffic toward a single BGP prefix over one of the links (fast switching performs per-destination-prefix load sharing). With CEF, the load is distributed based on the source-destination IP address pair combinations. Even if multiple clients send the traffic toward the same server, the load is spread across available links.
Generate HTTP(S) requests from Tcl shell
A few days ago, a reader sent me an e-mail titled “Telnet Automation from a Cisco Router” and complained that IOS Tcl does not support the expect commands (spawn, send and expect). Since Expect is a Tcl extension, not part of the core Tcl, it’s not included in Cisco IOS, which was the only answer I could give.
Can brain dumps be stopped?
Brain dumps are the biggest threat to the certification industry these days, significantly devaluing certifications that rely primarily on multiple-choice answers. Similarly to the threat-prevention measures adopted by airport security (read the insightful analysis of their behavior from Bruce Schneier, a renowned security guru), IT vendors are responding with high-tech measures.
Extranet with Overlapping Addresses
The idea to write an article describing how you can use MPLS VPN-enabled NAT to implement flexible extranets that allow participants to retain their existing (and sometimes overlapping) IP address space has been sitting in my to-do list for over a year.

After I’ve finally written it (without even hinting what I’ve been working on), I got several e-mails from my readers asking the questions this article answers, so it looks like the topic has suddenly become very hot. Do you have any ideas why that would be the case?
You’ll find the original article somewhere in this list.
Test the real-life skills of your job candidates
Numerous companies use certifications to screen job candidates. Even if all the caveats associated with this process are given, you might encounter candidates who have multiple high-level certifications but cannot differentiate a router from a box of cheese. How can you identify (and reject) such people?
Is it wise to use certifications in the candidate selection process?
My previous certification-related post described how some companies use certifications to filter job applicants for networking-related positions. Should you follow that example? If you’re in a country with a saturated job market, where the number of applicants far exceeds the number of job postings (consider yourself very lucky if you’re an employer), you should certainly use whatever filters you can to screen the hundreds of applications you receive … but be aware that you have potentially lost a few gems hidden in the flood.
MPLS support on 1800-series routers
Christoph sent me an interesting question a few days ago:
I played a bit arround with 2 Cisco 1803 and I found MPLS related configurations commands in IOS 12.4(15)T (Advanced Enterprise) on this box. MPLS was not listed as a included fearture in the Cisco Feature Navigator for this image and some searching at cisco.com took me to a 2 year old document telling me that MPLS isn't supported on this series. Some more searching took me back to the Cisco Feature Navigator which lists MPLS as feature for the Cisco 1805 router (which uses the same IOS image, afaik).
So, I'm a bit confused now if MPLS is really working / supported on the low-end Cisco ISR 1800 fixed series?
MPLS was mostly available but never supported on low-end platforms (including Cisco 2600). In those days I've taken some heat for reusing existing 2600-based labs to teach Cisco-internal MPLS courses (since we were teaching the students to configure unsupported devices :).
Anyhow, the "not supported" means exactly that: it may be available (well, it is), it may work (it actually does), but if it's broken (and I've seen at least one low-end-platform-specific bug in the early days) you can't complain.
Is anyone aware whether the official support for the MPLS on 1800 series has changed? If so, please share your information with us.
If you need to offer a production-grade service to your customers, don't use unsupported equipment; if you need a solution for your personal needs or you're building a lab, go ahead.
Why would I need a Web Application Firewall?
If you have been visited by a friendly Cisco sales engineer recently, you might have already heard about the ACE Web Application Firewall (WAF). If you’re curious enough to start investigating on your own, you might have stumbled across the WAF product description on Cisco’s Web site, which tells you … nothing.
Let’s start with an easy question: if I already have a firewall, why would I need another box with “fire” and “wall” in its name? The short answer is “Because Web programmers rarely know how to write secure Web applications.”
Book review: Voice over IP Security
Based on the title, I would assume that the Cisco Press book Voice over IP Security: Security best practices derived from deep analysis of the latest VoIP network threats attracts primarily senior voice engineers who know that they have to secure their production networks. The author of the book strongly disagrees with my opinion, however, spending more than a third of the book on baseline explanations of VoIP, SIP, H.323, firewalls, NAT, DES, IPSec…. I enjoyed the overview chapters, as I last configured VoIP before SIP was invented, but an experienced VoIP engineer would be disappointed.
This is QoS; Who Cares about Real-Time Response?
It all started with a innocuous question: can you detect voice traffic with EEM? Looks simple enough: create a QoS class-map that matches voice calls and read the cbQosClassMapStats
table in the CISCO-CLASS-BASED-QOS-MIB. The first obstacle was finding the correct indexes, but a Tcl script quickly solved that; I was ready to create the EEM applet. The applet failed to work correctly and after lots of debugging I figured out the counters in the cbQosClassMapStats
table change only every 10 seconds.
I couldn’t believe my eyes and simply had to test other MIB variables as well. As expected, the IF-MIB (standard interface MIB) counters increase in real-time, but obviously someone had the bright idea that we need to detect changes in traffic profile only every now and then. Although I've received numerous suggestions from my readers, none of them works on a Cisco 1800 or a Cisco 7200. Oh, well, Cisco developers from the days when I started working with routers would have known better…
… updated on Tuesday, November 17, 2020 11:51 UTC
The most convoluted MIB I’ve seen
Jared Valentine sent me a really interesting problem: he would like to detect voice traffic and start shaping TCP traffic for the duration of the voice call. The ideal solution would be an EEM applet reacting to the changes in the CISCO-CLASS-BASED-QOS-MIB; one of its tables contains the amount of traffic for each class configured in a service policy.
The MIB navigation looks simple: you just read the values from the cbQosClassMapStats
table, indexed by policy ID and class ID. The real problem is finding the correct index values. I could walk the MIB manually with a MIB browser or snmp_getnext TCL calls, but this approach is obviously not scalable, so I wrote a script that walks through the cbQosServicePolicy
, cbQosObjects
, cbQosPolicyMapCfg
and cbQosClassMapCfg
tables and prints the index values you need.
This script traverses the Class-based QoS MIB and displays service policies and classes attached to individual interfaces. The policy index and class index values are printed next to the policy/class name to help the operator fetch the desired SNMP variable from the statistics tables of the CISCO-CLASS-BASED-QOS-MIB.
Installation
- Download the source file into flash:cbindex.tcl
- Configure alias exec cbindex tclsh flash:cbindex.tcl
- Configure persistent CBQoS indexes with the snmp mib persist cbqos (otherwise the indexes will change after the router reload).
Usage guidelines
Usage: cbindex community
Command line parameters:
- Community: SNMP community with R/O access to the CISCO-CLASS-BASED-QOS-MIB
Source code
#
# title: Displays MQC class map indexes
# name: cbindex.tcl
# desc: The script traverses the Class-based QoS MIB and
# displays service policies and classes attached to
# individual interfaces. The policy index and class
# index values are printed next to the policy/class
# name to help the operator fetch the desired SNMP
# variable from the statistics tables of the
# CISCO-CLASS-BASED-QOS-MIB.
#
proc snmpInit { oid } {
global snmpCommunity
set getResult [ snmp_getnext $snmpCommunity $oid ]
if { [ regexp {snmp error} $getResult ] } {
puts "SNMP calls with community $snmpCommunity fail"; return 0
}
if { [ regexp {oid='(.*)'} $getResult ignore nxtoid ] } {
if { [string first $oid $nxtoid] == 0 } { return 1 }
}
puts "MIB $oid not implemented in this IOS release"; return 0;
}
proc snmpGet { oid result } {
global snmpCommunity
upvar $result r
if { [info exists r] } { unset r }
set getResult [ snmp_getone $snmpCommunity $oid ]
if { [ regexp {snmp error.*text='(.*)'} $getResult ignore errtxt ] } {
error "snmpGet - $errtxt"; return 0
}
if { [ regexp {oid='(.*)'.*val='(.*)'} $getResult ignore oid result ] } {
if { ! [ string equal $result "NO_SUCH_INSTANCE_EXCEPTION" ] } {
set r(OID) $oid ;
set r(VALUE) $result ;
return 1;
}
}
return 0;
}
proc snmpGetNext { oid result } {
global snmpCommunity
upvar $result r
if { [info exists r] } { unset r }
set getResult [ snmp_getnext $snmpCommunity $oid ]
if { [ regexp {snmp error.*text='(.*)'} $getResult ignore errtxt ] } {
error "snmpGet - $errtxt"; return 0
}
if { [ regexp {oid='(.*)'.*val='(.*)'} $getResult ignore oid result ] } {
if { ! [ string equal $result "NO_SUCH_INSTANCE_EXCEPTION" ] } {
set r(OID) $oid ;
set r(VALUE) $result ;
set oidSplit [ split $oid "." ]
set r(NAME) [ lindex $oidSplit 0 ]
set r(INDEX) [ lreplace $oidSplit 0 0 ]
set r(IDXLIST) [ join $r(INDEX) "." ]
return 1;
}
}
return 0;
}
proc snmpGetInTable { oid result { parentoid "" }} {
global snmpCommunity
upvar $result r
snmpGetNext $oid r
if { ! [info exists r(OID)] } { return 0 }
if { [string equal $parentoid ""] } {
set oidSplit [ split $oid "." ]
set parentoid [lindex $oidSplit 0]
}
if { [string first $parentoid $r(OID)] != 0 } { return 0 }
return 1;
}
proc printQosClassIndex {} {
global snmpCommunity
set oid "cbQosIfIndex"
array set dirLookup { 1 in 2 out }
set cnt 0
while { [ snmpGetInTable $oid svcPolicy ] } {
if { [snmpGet "ifDescr.$svcPolicy(VALUE)" ifDescr] } {
snmpGet "cbQosPolicyDirection.$svcPolicy(INDEX)" svcDirection
snmpGetNext "cbQosConfigIndex.$svcPolicy(INDEX)" policyObject
snmpGet "cbQosPolicyMapName.$policyObject(VALUE)" policyName
puts "\n$ifDescr(VALUE) ($dirLookup($svcDirection(VALUE))): $policyName(VALUE) ($svcPolicy(INDEX))"
set coid "cbQosObjectsType.$svcPolicy(INDEX)"
set parentoid $coid
while { [ snmpGetInTable $coid svcClass $parentoid ] } {
if { $svcClass(VALUE) == 2 } {
snmpGet "cbQosConfigIndex.$svcClass(IDXLIST)" svcClassConfig
snmpGet "cbQosCMName.$svcClassConfig(VALUE)" svcClassName
puts " $svcClassName(VALUE) $svcClass(IDXLIST)"
}
set coid $svcClass(OID)
}
} else { error "Cannot get interface name for service policy $svcPolicy(VALUE)" }
set oid $svcPolicy(OID)
}
}
set snmpCommunity [lindex $argv 0]
if { [string equal $snmpCommunity ""] } { set snmpCommunity "public" }
if { ! [ snmpInit "cbQosObjectsType" ] } return
printQosClassIndex
Sample usage scenario
The following QoS classes and policies have been configured on the router:
class-map match-all Mail
match protocol smtp
!
class-map match-all Web
match protocol http
!
class-map match-all SecureWeb
match protocol secure-http
!
class-map match-any Surfing
match class-map Web
match class-map SecureWeb
!
class-map match-all Files
match protocol ftp
!
policy-map Internet
class Web
bandwidth 128
class SecureWeb
priority 64
class Mail
bandwidth 32
!
policy-map MailOrFtp
class Mail
set ip precedence 0
class Files
set ip precedence 0
class Surfing
police 16000
class class-default
police cir 8000
exceed-action drop
!
interface Serial1/0
service-policy input MailOrFtp
service-policy output Internet
!
interface Serial1/1
service-policy output MailOrFtp
The cbindex script reported the following SNMP indexes:
c7200#cbindex Test
Serial1/0 (in): MailOrFtp (48)
Web 48.383777
Surfing 48.1970017
Mail 48.4297921
Files 48.13110129
class-default 48.14779377
SecureWeb 48.15077857
Serial1/0 (out): Internet (50)
Mail 50.10516033
Web 50.14007809
SecureWeb 50.14520625
class-default 50.15008753
Serial1/1 (out): MailOrFtp (66)
Web 66.383777
Surfing 66.1584993
Files 66.4236097
Mail 66.11615889
SecureWeb 66.15077857
class-default 66.15082481
Based on these indexes, you could monitor the bit rate of the Web class in outbound policy configured on Serial 1/1 with SNMP variable cbQosCMPrePolicyBitRate.66.383777
.
c7200#tclsh
c7200(tcl)#snmp_getone Test cbQosCMPrePolicyBitRate.66.383777
{<obj oid='cbQosCMPrePolicyBitRate.66.383777' val='0'/>}
Certifications: a new barrier to entry
Recent blog posts indicate that, in at least some market segments, IT certifications are becoming a new barrier to entry: companies require a specific set of certifications in their job offerings and use those requirements to filter the candidates who are invited to the initial interview. Obviously, IT vendors pushing the certifications are getting some real traction. On the other hand, anecdotal evidence indicates that certification holders are sometimes able to memorize vast amounts of information without being able to put it to use (I don’t want to imply that they used other, less honest methods).