Blog Posts in November 2007

BGP Fast Session Deactivation

We all know that BGP is meant to converge slowly… well, the MPLS/VPN service providers tend to disagree, as their users are not used to minute-long convergence times. One of the major components of slow BGP convergence is the time it takes a router to discover that a neighbor has disappeared. Traditionally, the BGP keepalive packets were sent every minute and it took up to three minutes to discover that a neighbor is down.

Of course you could fine-tune those times with the neighbor timers configuration command, but the reduced timers resulted in increased TCP traffic and consequently increased CPU load, which could reach tens of percents if the timers were set to a few seconds and the router had lots of BGP neighbors.

read more see 2 comments

Configure DNS Servers Through IPCP

After I've fixed the default routing in my home office, I've stumbled across another problem: the two ISPs I'm using for my primary and backup link have DNS servers that reply solely to the DNS requests sent from their own IP address range:

When the traffic is switched from the primary to the backup ISP, I therefore also need to switch the DNS servers. Fortunately, this is quite easy to do on a router; you just need to configure ppp ipcp dns request on the dialer interface and the router starts asking for the DNS server address as part of the IPCP negotiation.

read more see 8 comments

Enhanced show interfaces command

It's amazing how many options (most of them still undocumented) the show interfaces command accepts in IOS release 12.4T (I won't even start guessing when each one was introduced, if you're running old IOS releases, please feel free to comment):

  • show interfaces description displays interface names, L1 and L2 status (line and line-protocol status) and interface description. Extremely handy if you want to check which interfaces are up/down.
  • show interfaces counters protocol status displays the L3 protocols active on each interface.
  • show interfaces summary displays the state of various interface queues and related drop counters in a nice tabular format.
  • show interfaces accounting displays per-protocol in/out counters.

Here are a few sample printouts:

read more see 9 comments

Can I combine EEM applets with Tcl shell?

When I’ve been describing the limitations of kron, someone quickly asked an interesting question:

As I cannot insert extra input keystrokes with EEM applet, can I run a Tcl script from it with the action sequence cli command “tclsh script” command and use the typeahead function call to get around the limitation?”

The only answer I could give at that time was “maybe” … and obviously it was time for a more thorough test. The short result is: YES, you can do it (at least in IOS release 12.4(15)T1).

read more see 16 comments

Kron: poor-man's cron

When two groups within Cisco needed time-based command execution in Cisco IOS, they (in a typical big-corporation fashion) decided to implement the same wheel from two different sets of spokes and rims. One group built the Embedded Event Manager with its event timer cron command (introduced in 12.2(25)S and 12.3(14)T), the other group created the more limited kron command set (introduced in 12.3(1)).

read more see 5 comments

Enable password or enable secret?

I've stumbled across a blog post that indicates there's still confusion on some fundamental configuration issues. I will not even try to guess whether there is a wide consensus on how to configure a router, but these are the facts (and here is a ten year old position from Cisco):
see 11 comments

Emulate dialup links with serial lines

I had to figure out various PPP parameters (and associated Cisco IOS behavior) and didn't have real dial-up equipment in my lab setup. I could have gone with PPPoE, but it turned out it's way simpler to emulate dialup connections (at least the PPP negotiations work as expected) on fixed serial lines. This is the minimum setup you need on the “caller” side …
interface Serial1/0
 ip address negotiated
 encapsulation ppp
 ppp authentication pap optional
 ppp pap sent-username client password 0 client
… and this is the “server”-side configuration:
interface Serial1/0
 ip address 10.0.0.33 255.255.255.252
 encapsulation ppp
 peer default ip address 10.0.0.34
 ppp authentication pap callin
!
username client password client
To trigger PPP negotiations, shut down and re-enable the serial interface on either side.

Note: As I'm using PAP authentication, I could use the more secure username secret configuration command, which would not work with CHAP.

see 8 comments

IPv6 Deployment: Time for Action?

A while ago I was asked to write an article about IPv6 training. I could just cover the training aspect, like what’s offered (answer: not much) and whether someone can train the whole operations team like you could in the IPv4 or MPLS/VPN world (answer: no), but I wanted to understand whether anyone is really using IPv6 in a production network.

I found a few academic networks (after all, there are about 2000 IPv6 prefixes assigned and someone should be doing something with them), but not much of what I would call a real production environment, which is a bad thing, as it looks like the IPv4 address space will get saturated in a few years.

read more see 8 comments

Type 7 decryption in Cisco IOS

Tim Riegert sent me an interesting hint: you don't need password crackers to decode type-7 passwords, you just need access to a router. Here's how you do it:

We'll turn on type-7 encryption for local passwords and generate a test username

R1(config)#service password-encryption
R1(config)#username test password t35t:pa55w0rd

Next we'll inspect the generated username with the show running command

R1(config)#do show run | include username
username test password 7 08351F1B1D431516475E1B54382F

Now we'll create a key chain and enter the type-7 encrypted password as the key string …

R1(config)#key chain decrypt
R1(config-keychain)#key 1
R1(config-keychain-key)#key-string 7 08351F1B1D431516475E1B54382F

… and the show command does the decryption for us.

R1(config-keychain-key)#do show key chain decrypt
Key-chain decrypt:
    key 1 -- text "t35t:pa55w0rd"
        accept lifetime (always valid) - (always valid) [valid now]
        send lifetime (always valid) - (always valid) [valid now]
see 12 comments

Show active IOS processes

You can use the show process cpu sorted command in combination with an output filter to display only those IOS processes that consumed noticeable amount of CPU time in the last five minutes, last minute or last five seconds. Use the following patterns to construct your regular expression:
  • The [0-9.]+% pattern will match any non-zero percentage;
  • The 0.00% pattern will obviously match the zero-percentage display;
  • As the percentage figures are separated by various amounts of whitespace characters, we have to use the ' +' pattern to match those;
The show filter should exclude the processes that have the zero percentage in the desired column and any percentage in the other two columns (any other filter would show too many or too few processes). To display processes active in the last minute, use the show process cpu sorted 1min | exclude [0-9.]+% +0.00% +[0-9.]+% command (and define an alias to make it easier to use).
read more add comment

Persistent EEM variables

Someone has asked me a while ago whether it's possible to retain variable values between invocations of an EEM policy. Since a new copy of Tcl interpreter is started for each event, global variables obviously won't work; they are lost as soon as the Tcl policy is finished. A potential solution is to modify the router's configuration and save the values you wish to preserve in event manager environment, but that's a time-consuming process that interferes with whatever router configuration management process you have.

The real solution is based on the appl_setinfo and appl_reqinfo calls. They work, but like many other Tcl-related IOS features they are … well … weird.
read more add comment

Ones Are Slower than Zeroes

Thinking about the implications of bit stuffing I wrote about in the SDLC post, I realized that long sequences of ones would be transmitted slower than long sequences of zeroes due to an extra bit being inserted after every fifth consecutive one. The theory would predict a 20% decrease in transmission speed.

Of course I wanted to test this phenomenon immediately. I connected two routers with a low-speed (64 kbps) link, and started a series of pings. Not surprisingly, the results confirmed the theory:

read more see 3 comments

Catch Skype with Flexible Packet Matching

Joe Harris published an excellent post detailing how you can use Flexible Packet Matching to recognize (and potentially block) Skype traffic. The solution depends on recognizing the first four bytes sent by the Skype application in a TCP session. While this is a great idea, you have to be aware that there's always a non-zero chance of false positives, more so as the described filter is testing the beginning of the payload in every TCP packet (not just the first data packet in the session).
see 3 comments

Back to the roots: it all started with SDLC

My recent post about problems with old modems has generated a lot of comments with some very useful ideas, but nobody addressed the question “why was a long string of ones not a problem?”, so let's start there. Almost all WAN synchronous protocols in use today are descendants of venerable SDLC invented by IBM more than 30 years ago. SDLC was later extended to support connectionless and balanced modes, resulting in HDLC. PPP is just an extension of HDLC, adding support for negotiations and standard layer-3 protocol demultiplexing. In SDLC, IBM also solved the frame delimiting and associated escape character problem inherent in previous protocols like BSC (DLE was used in BSC) by introducing bit stuffing: a zero would be inserted after five consecutive ones (and silently removed by the receiver) to differentiate the regular data stream from framing (six consecutive ones) and abort (more than six consecutive ones) sequences. Thus, the HDLC (or PPP) data stream can never contain more than six consecutive ones and the long sequences of ones never cause synchronization loss.

IBM obviously also had problems with bad modems and solved it with the NRZI encoding that was part of SDLC standard (and a major pain in the good old days when the appliques on the old Cisco routers did not support it and we've been trying hard to penetrate IBM accounts). You can still configure NRZI encoding on most routers' serial links (it might depend on the actual hardware platform) with the nrzi-encoding interface configuration command (you had to do it with jumpers in the AGS+). Incidentally, changing interface encoding to NRZI was really helpful when you had to break things in the preparation for the troubleshooting part of the original CCIE lab).

Enough theory, let's summarize the proposed solutions:
  • The nrzi-encoding (if available) is the best one, as it reliably solves the problem, is transparent and does not incur additional overhead.
  • Compression or encryption are OK, but they result in significant CPU overhead (unless you have hardware encryption/compression modules) and might (at least in theory) still produce a long sequence of zeroes, although with a very low probability. IPSec also introduces overhead due to additional IPSec headers.
  • LFI (effectively multilink PPP over a single link) is also a good solution, as the PPP framing and MLPPP headers break the long sequences of zeroes (you might have to fine-tune the fragment size with ppp multilink fragment size configuration command), but it introduces overhead on the WAN link.
  • IP fragmentation would work, but would be quite bandwidth-consuming. If the fragmentation would be performed by the router, the overhead would be 20 bytes per fragment (IP header), if the sending host performs the fragmentation, the overhead is 40 bytes per fragment for TCP sessions. For example, if we reduce the IP MTU size to 256 bytes, the TCP session overhead is over 18% (and we were scoffing at the ATM designers that made us live with 10% overhead).
There were also a few suggestions that would not work very well:
  • The invert data command would only help if the modem has problems with long strings of zeroes, not with long strings of the same value.
  • The tunnel key command just sets a 4-byte field in the GRE header but does not affect the encapsulated data at all.
see 3 comments

React to excessive jitter with EEM

William Chu sent me a working configuration he uses to measure jitter with the IP SLA tool and react to excessive jitter on the primary link. First you have to create the jitter probe with the IP SLA commands:

ip sla monitor 3000
 type jitter →
   dest-ipaddr 199.11.18.168 dest-port 12333 →
   source-ipaddr 199.11.18.169 codec g729a →
   codec-numpackets 100
 tos 184
 frequency 10

Note: The continuation character (→) indicates that the configuration command spans multiple lines

Next you have to define the IP SLA reaction to excessive jitter. William configured his router to react when the jitter exceeds 300 milliseconds and returns back to normal when the jitter falls below 290 milliseconds (some hysteresis is always a good thing).

ip sla monitor reaction-configuration 3000 →
  react MOS threshold-value 300 290 →
  threshold-type consecutive →
  action-type trapOnly

As the last step in the SLA configuration, you have to start the probe:

ip sla monitor schedule 3000 →
  life forever start-time now

After the SLA probe and out-of-bounds reaction have been configured, the router will generate syslog messages whenever the jitter gets above the threshold as well as when it falls below the second threshold. You can then use the EEM applets to act on the syslog messages:

event manager applet MOS-Below
 event syslog occurs 1 period 120 →
   pattern "Threshold below for MOS"
 ... actions ...
!
event manager applet MOS-Above
 event syslog occurs 1 period 120 →
   pattern "Threshold exceeded for MOS"
 ... actions ...

see 4 comments

For the oldtimers: swamped with zeroes

In the pre-DSL days, you had two options to get a short-haul high-speed link (at least in Europe): take E1 (or fractional E1) from a telecom (which was more expensive than a highway robbery, as the cost was recurring) or use baseband modems with proprietary encoding techniques on physical copper wires (assuming you could get them). As it turned out, some of these encoding techniques were not as good as the others (but the equipment was relatively cheap, so the budget limits usually forced the decision). We had our own share of modem-related problems, but they were never as bad as what I've heard from one of my students: his modems would lose synchronization when transmitting a long string of zeroes over a regular synchronous serial interface; ping ip 1.2.3.4 size 1000 data 0000 would be enough to bring down the link.

And now two questions for you:
  • What could you do on the router to fix this problem?
  • Why was the synchronization retained when transmitting a long string of ones?
see 11 comments

Download router configurations via TFTP

In a previous post, I've described how you can turn your router into a TFTP server. As you can configure the router to serve any file residing on it, you can also pull startup and running configuration from it with TFTP, providing that you configure:
tftp-server nvram:startup-config
tftp-server system:running-config

Warning: Due to total lack of any security features in TFTP protocol, use this functionality only in lab environment.

see 5 comments
Sidebar