Building network automation solutions

9 module online course

Start now!

Category: you've asked for it

Things you cannot do with Tclsh

What would you think if you’d receive three queries about the same (somewhat obscure) feature within six hours? It started with a nice e-mail from an engineer that I’ve corresponded with in the past. He wanted to send a Wake-on-LAN packet to a PC in a remote office. Usually you could use the ip directed-broadcast feature, but he wanted to use the remote office router to generate the packet.

read more see 8 comments

Interactions between IP routing and QoS

One of my readers sent me an interesting question a while ago:

I reviewed one of your blog posts "Per-Destination or Per Packet CEF Load Sharing?" and wondered if you had investigated previously on how MQC QoS worked together with the CEF load-sharing algorithm (or does it interact at all)? For example, let's say I have two equal cost paths between two routers and the routing table (as well as CEF) sees both links as equal paths to the networks behind each router. On each link I have the same outbound service policy applied with a simple LLQ, BW, and a class-default queues. Does CEF check each IP flow and make sure both link's LLQ and BW queues are evenly used?

Unfortunately, packet forwarding and QoS are completely uncoupled in Cisco IOS. CEF performs its load balancing algorithm purely on source/destination information and does not take in account the actual utilization of outbound interfaces. If you have bad luck, most of the traffic ends on one of the links and the packets that would easily fit on the other link will be dropped by the QoS mechanisms.

You could use multilink PPP to solve the problem in low-speed environments. With MLPPP, CEF sends the traffic to a single output interface (the Multilink interface) and the queuing mechanisms evenly distribute packet fragments across the links in the bundle.

In high-speed environments, you can only hope that the number of traffic flows traversing the links will be so high that you’ll get a good statistical distribution (which is usually the case).

see 3 comments

Flash-based DHCP database

Pete sent me an interesting question a while ago:

It might be interesting to write an article about ip dhcp database flash:dhcp-db command, documenting the pros of surviving a reboot versus cons of wear on the flash device.

I’ve already written about a few problems that can be solved with the DHCP database (but obviously a longer text is warranted … already stored in my to-do list) and it took me a while to find the time to dig out the relevant information on the flash device wear.

read more see 5 comments

This is QoS; Who Cares about Real-Time Response?

It all started with a innocuous question: can you detect voice traffic with EEM? Looks simple enough: create a QoS class-map that matches voice calls and read the cbQosClassMapStats table in the CISCO-CLASS-BASED-QOS-MIB. The first obstacle was finding the correct indexes, but a Tcl script quickly solved that; I was ready to create the EEM applet. The applet failed to work correctly and after lots of debugging I figured out the counters in the cbQosClassMapStats table change only every 10 seconds.

I couldn’t believe my eyes and simply had to test other MIB variables as well. As expected, the IF-MIB (standard interface MIB) counters increase in real-time, but obviously someone had the bright idea that we need to detect changes in traffic profile only every now and then. Although I've received numerous suggestions from my readers, none of them works on a Cisco 1800 or a Cisco 7200. Oh, well, Cisco developers from the days when I started working with routers would have known better…

read more see 14 comments

Interactive online sessions: your input is highly appreciated

In mid-December, I’ll do my first IOS Hints Online Session. These sessions will be short (30-60 minutes), very interactive (I hope, but that’s your choice) and focused on an interesting design/deployment aspect. The description of the design/deployment challenge addressed by the session will be available well in advance at the time when you’ll be able to register.

Each session will start with a few diagrams explaining the proposed solution to the session’s topic and continue with hands-on explanation on actual devices. Each session will be limited to ~15 participants who will be able to actively participate, ask questions, propose alternative solutions or even discuss their actual issues (assuming they are somewhat related to the primary topic of the session).

I have a “few” ideas what could be covered in these sessions, but having a real-life challenge coming from the readers of my blog would be much better. If you have a good idea that could fit into this concept, please send me a short description before Friday, November 21st. I’ll collect the best ones, publish short descriptions in a blog post and you’ll prioritize which ones you’d like to see first.

see 3 comments

Load balancing quirks

One of my readers has noted an interesting load-balancing behavior: when he was running traceroute tests from various routers in a topology similar to the one displayed below, the traceroute outputs indicated per-packet load balancing (both paths were used) when they were initiated from R2 or R3, but used a single path when initiated from R1 or R4.
The reason for this behavior is very simple: if you do traceroute from R1 to R4, R2 and R3 perform CEF switching, which usually does load balancing based on source-destination IP address pairs, so all probe packets from R1 to R4 travel along the same path. If you start traceroute from R2 or R3, the packets are process-switched on the first hop (from R2 to R3, for example) and thus alternate between the parallel links.

This article is part of You've asked for it series.

see 5 comments

SNMP v3 users not shown in running-config

Ralf sent me a SNMPv3 question:
If I create a SNMPv3 user which has a password (snmp-server user userthree groupthree v3 auth md5 user3passwd), this user does not appear in the running- or startup-config. Cisco even documents this if you know what to look for.

I strongly suspect (although I did not test this) that these users are also missing from configuration exported to TFTP servers. What would be the recommended way to make usable config backups of routers with such users?
Like certificates, the SNMPv3 users are stored in private-config and thus never appear in the router configuration. If you want to have a backup of the user data, create a text file on one of your NMS servers, add SNMPv3 usernames and passwords in the text file and use the copy somewhere running-config to configure SNMPv3 users on the routers.
This article is part of You've asked for it series.
see 4 comments

IOS scheduling parameters

Peter Weymann sent me a really intriguing question:

A few days ago I started reading the Ciscopress book End-to-End Network Security: Defense-in-Depth and stumbled over the scheduler command. This one could be used to allocate time that the cpu spends on fast switching packets or process switching packets, if I understand it correctly. They also mention interrupting CPU processes but honestly I don't really understand how it works.

Cisco routers support (at least) three forms of layer-3 switching (formerly known as routing). CEF switching and fast switching are performed entirely within the interrupt context (I/O adapter interrupts a process the CPU is currently executing and all the work is done before the process resumes). Process switching is performed in two steps: packet is briefly analysed within the interrupt context and requeued into the IP Input process where it's eventually switched. Almost all I/O adapters used these days use a concept of RX/TX rings to communicate with the CPU, meaning that the CPU potentially has to handle more than one packet for each interrupt.

Fast switching is gone starting with IOS release 12.4(20)T.

Under very high load, the packet arrival rate could be so high that the router would constantly service packets within the interrupt context without ever returning back to the IOS processes.

You can check the CPU load incurred by the interrupt context and IOS processes with the show process cpu command. The second number in the five seconds part of the first line tells you the amount of interrupt context activity in the last five seconds.

To prevent the starvation of IOS processes (which could result in keepalive and routing protocol problems, eventually leading to loss of routing protocol neighbors), the scheduler allocate command limits the amount of time that can be spent in the interrupt context and allocates some guaranteed time to the IOS processes. Very probably the routers have a mechanism to mask the requests from the I/O adapters during that period so that the CPU is not interrupted (BTW, this slightly increases the jitter).

A similar command is the scheduler interval command. IOS has high- and low priority processes. Whenever the CPU has to decide what process to run (usually following an interrupt or when a process decides it's done with its work), it will run a high-priority process if one is ready. This could lead to starvation of low-priority processes and the scheduler interval command specifies the maximum amount of time the higher-priority processes can consume before a low-priority process is given a chance to run.

Unless you have serious (and I mean __serious__) problems in your network, don't play with these commands. They are a last-resort things you can do if you're under very heavy load and still need access to the exec to reconfigure the router. In most cases, you should not have to worry ... and anyhow, if the CPU load is close to 100%, you have other problems anyway.

Apart from the Inside Cisco IOS Software Architecture book that you absolutely must have if you're interested in (a bit outdated) view of the internals of Cisco IOS, you can get more information in these documents:

see 1 comments

Is a label imposed in case of Penultimate Hop Popping?

Shivlu Jain sent me an interesting question:
I'm wondering whether a router performing penultimate hop popping (PHP) imposes an IGP label or not.The value of implicit null is 3; does it mean the router imposes this label (and adds four bytes to the packet)?
The penultimate router does not impose the IGP label (that's why this behavior is called penultimate hop popping). However, the egress router has to signal to its upstream neighbor (the penultimate router) that it should NOT impose a label, so it uses "implicit null" label (= 3) in TDP/LDP updates to signal that the top label should be popped, not rewriten.
This article is part of You've asked for it series.
see 2 comments

Measure the cable lengths on a Catalyst switch

Ken McCoy sent me a short question:
At one point someone posted an article about a command you could run on the Catalyst switch that would give you back the distance of the cable between the switch and end device, but now I can't find it.
I remembered reading the same article and after I've figured out the underlying technology is called TDR (Time Domain Reflectometer), uncle Google immediately provided a reader tip from Csaba Farkas.
This article is part of You've asked for it series.
see 5 comments

The “fallback global” VRF option does not exist in Cisco IOS

Cheng sent me an interesting question:
I'm reading your book MPLS and VPN Architectures and I've found the ip vrf forwarding name fallback global command in the “Additional Lookup in the Global Routing Table” section. I can only find this command in Junos, but not in IOS.

… and he was right. When we were writing the book, we described several features that were still in development as it looked like they would be in the production code by the time the book was published. Many of them made it into the public IOS releases (for example, the Carrier's Carrier architecture), but some of them (like this command) simply vanished from the surface.

However, it looks like the engineers that switched from Cisco to Juniper took the concept with them and implemented it in JunOS, so JunOS has this feature but IOS doesn't.

This article is part of You've asked for it series.

see 1 comments

When “copy” actually means “merge”

Marcus Jensen asked me a very interesting question:

I want to send 3 lines of configuration to a remote router, but I know the first line will kill my connection. Can I save these 3 lines of code to a text file, and then issue a Tcl command to add those to the running config?

The solution is much simpler and does not have to involve Tcl at all. The copy something system:running-config command merges the configuration commands in the source file with the current running configuration.

You can store the configuration commands you want to execute in a local file (even in NVRAM) or you could execute them directly off a file server (using HTTP, FTP, TFTP or SCP protocol).

This article is part of You've asked for it series.

add comment

Interesting QoS problem on Catalyst 3750

Mohammad Faraz Rehan has encountered an interesting problem when using IP access-list based class/policy maps on Catalyst 3750:

When I try to apply the same service-policy to more than 15 interfaces, the configuration command fails and the switch generates the following syslog message:

%QOSMGR-4-POLICER_PLATFORM_NOT_SUPPORTED: Policer configuration has exceeded hardware limitation for policymap …

I've tried to help him with various TCAM-related information and in the end he found an interesting solution to the problem:

It looks like there is a limit related to using the same access-list/class-map/policy-map on multiple interfaces.

The first time I was applying the same policy-map (19 classes/19 ACLs/46 ACL lines) on all interfaces, but the switch would not accept it on more than 15 interfaces. Another test scenario had 18 classes/18 ACLs/52 ACL entries and the same policy-map would only work on 13 interfaces.

Now we implemented 24 different policy-maps, class-maps and ACLs remained the same, and the switch is happy.

see 1 comments

OSPF default route based on IP SLA

Olivier Guillemain has asked an interesting question: “how could I originate a default route into OSPF based on IP SLA (for example, based on pinging a remote IP address)?

This is very easy to do when the router originating the default route into OSPF needs an SLA-based default route itself:

  1. Configure IP SLA and a corresponding track object;
  2. Configure a default route using reliable static routing
  3. Advertise the default route into OSPF with the default-information originate router configuration command

The solution is a bit more complex when the router originating the default route into OSPF should not have a default route. In this case, you could use a routing trick:

  1. Configure IP SLA and a corresponding track object as before;
  2. Use reliable static routing to configure a static host route for a bogus IP address (for example, pointing to null0 (for example, ip route null 0 track 100). Obviously this host route should not be redistributed into any routing protocol.
  3. Conditionally advertise default route into OSPF based on presence of the static host route.

Note: This article is part of You've asked for it series.

see 1 comments

mturoute: ping-mode output

Jeff West has asked me to document the printout produced by the mturoute utility. Here's the first part of the documentation.

mturoute works in two modes:
  • Without the -t flag, it sends variable-lenght ICMP echo packets to the specified destination address, trying to figure out the largest packet that is successfully propagated to the destination.
  • With the -t flag, it uses traceroute-like algorithm to find the hop-by-hop IP addresses (the source IP addresses of the ICMP TTL exceeded replies) and uses the same packet-size-calculating algorithm to measure the path MTU to each hop.

Today we'll focus on the non-trace mode. It tries to measure the path MTU with a bisection method varying the packet sizes between minimum MTU (92) and maximum MTU (specified with the -m parameter, default is 10000 bytes). The payload size of the first packet (without the -m flag) is thus 5046 bytes ((10000 + 92)/2).

On each iteration, the algorithm prints a “cryptic” sign indicating whether the ping with the current payload size succeeded. The following indications are given:

  • '+': ICMP echo reply arrived
  • '-': The ping failed (for various reasons, including exceeding the path MTU)
  • 'u': ICMP destination unreachable response arrived, indicating blackhole or access-list.
  • ICMP unreachable is considered a successful response; at least we're measuring the path MTU up to the failure point

  • '.': timeout. The ping will be retried up to three times (or the number specified with the -r option).
In the ping mode, mturoute prints the current ICMP payload size at each step, resulting in a printout similar to the one below. If you'd have specified the -d option, the printout would include detailed status codes from the IcmpSendEcho function.
$ mturoute
* ICMP Fragmentation is not permitted. *
* Maximum payload is 10000 bytes. *
- ICMP payload of 5046 bytes failed..
- ICMP payload of 2569 bytes failed..
+ ICMP payload of 1330 bytes succeeded.
- ICMP payload of 1949 bytes failed..
- ICMP payload of 1639 bytes failed..
- ICMP payload of 1484 bytes failed..
+ ICMP payload of 1407 bytes succeeded.
- ICMP payload of 1445 bytes failed..
+ ICMP payload of 1426 bytes succeeded.
+ ICMP payload of 1435 bytes succeeded.
+ ICMP payload of 1440 bytes succeeded.
+ ICMP payload of 1442 bytes succeeded.
+ ICMP payload of 1443 bytes succeeded.
+ ICMP payload of 1444 bytes succeeded.
+ ICMP payload of 1444 bytes succeeded.
Path MTU: 1472 bytes.

Note: To use the debug-enabled version of mturoute, or the version that does not need VC++ runtime, download the new ZIP archive from my web site.

see 3 comments