Your browser failed to load CSS style sheets. Your browser or web proxy might not support elliptic-curve TLS

Building network automation solutions

6 week online course

reserve a seat

Generating layer-2 broadcast from a regular IP packet

The Wake-on-LAN discussion we had a while ago brought us nowhere; there's simply no way to generate UDP packets on the router. I thought I could use Application Performance Monitor's Tcl scripts to generate the packet, but it looks like APM has been removed from recent IOS releases (and it's not clear whether you can use APM without a peer router).

The discussion nonetheless had an interesting side effect. Robert Turnšek sent me an interesting trick: with static ARP you can generate layer-2 broadcasts with a layer-3 unicast packet.

Let's assume your LAN has IP prefix and you want to use as the IP address that will generate MAC-layer broadcasts (you need to give this address to a WOL program). Configure arp FFFF.FFFF.FFFF and you're done.

You might wonder how this technique differs from directed subnet broadcast. The important difference is that although every IP host on the subnet will receive the layer-2 broadcast, they will ignore the packet since it's not addressed to them. This solution is thus not vulnerable to the smurf attack.

see 3 comments

Ah, the wonderful quoting process

Network World just published an article on LISP, including a quote attributed to Geoff Huston:

LISP relies on tunneling, and tunneling is not 100% perfect. At times, the tunnel passes a packet that's too big and it disappears without a trace to the sender or the recipient…That's really bad.

We all know what can go wrong with the tunnels: a combination of:

  • MTU setting on a core link that is too low to accomodate tunnel envelope+payload with
  • Tunnel headend that cannot respond to Fragmentation needed ICMP message.

I sincerely doubt that the authors of the LISP draft would not be aware of those problems (Cisco's gear is one of largest source of tunnels in the IP world) or that mr. Huston would not know what he's talking about (read the MTU articles on his personal blog).

The only explanation for the fairy-tale quote in the article is thus the process of multi-tier simplification until facts became adjusted to the technology awareness of someone in the publishing pipeline.

Add comment

Rate-limiting inbound traffic on DSL

Julian is faced with an interesting challenge:

In the real world, many customers using DSL solutions have their Internet connection disrupted by one internal user performing a large download. On a typical DSL solution, implementing quality of service on outbound traffic is trivial (you can use PQ, CBWFQ, policing or shaping). However, how does one rate-limit inbound traffic in a sensible fashion? Turnkey solutions like packeteer allow inbound classes of traffic like HTTP to be rate limited per flow by dynamically changing window sizes.

Cisco IOS has three basic QoS mechanisms: queuing, shaping and policing. It cannot intercept a TCP session and slow it down by reducing its window size (like PacketShaper).

A queuing mechanism works only if the (temporary) packet arrival rate is higher than the interface transmission rate and thus an output queue forms. When the router receives packets from a DSL link and sends them to an Ethernet link, the output queue remains almost empty and thus you cannot use any queuing mechanism.

Policing works regardless of the size of the output queue, but it cannot be fine-tuned to react to interface congestion (unless you use EEM interface triggers to detect high interface utilization). Furthermore, IOS does not have proportional policing of individual traffic flows.

The only remaining mechanism is shaping: you have to create an artificial bottleneck on the outbound (LAN) interface with the shape MQC command. The shaping rate should be less than the useful speed of your DSL link. The shape command introduces a bottleneck: the packets arrive through the DSL interface faster than they can be sent to the (shaped) LAN interface. Now you have an output queue that you can influence with further QoS policies (you need to configure hierarchical MQC). Without further configuration, the output queue created with the shape command uses per-flow weighted fair queuing, giving each layer-4 session a fair share of the bandwidth.

The output rate configured with the shape command has to be significantly smaller than the DSL rate, otherwise the output queue forms only sporadically. By shaping the inbound DSL packets on the LAN interface you reduce the maximum throughput, slightly increase latency and introduce jitter.

This article is part of You've asked for it series.

see 5 comments

ADSL overhead

Yesterday I’ve described the difference between line rate and bit rate (actually physical layer gross bit rate and physical layer net bit rate). Going to the other extreme, we can measure goodput (application-level throughput), which obviously depends on multiple factors, including the TCP window sizes and end-to-end delays. There are numerous tools to test the goodput from/to various locations throughout the world ( worked quite nicely for me) and you’ll soon discover that the goodput on your DSL line differs significantly from what the ISP is advertising.

Apart from obvious marketing ploys, the difference arises due to the baroque Russian dolls encapsulation stack used on ADSL (see the ADSL-based protocol stacks section in this document). A typical PPPoE connection runs IP over PPP (2 bytes) over PPPoE framing (6 bytes) over Ethernet framing (14 bytes) over RFC 1483 bridged connection (8 bytes) over AAL5 (8 bytes) over ATM cells (5 byte overhead for every 48 bytes of payload). Every IP packet is thus burdened with 38 bytes of DSL protocol stack overhead plus 10.4% ATM cell tax. Assuming the average IP packet size in the Internet is around 500 bytes, the DSL protocol stack overhead is 6%, resulting in a total of over 16% of lost bandwidth just to accommodate the committees that constructed this monster.

Update @ 2009-03-27: I forgot yet another encapsulation technique: AAL5. I also need to investigate whether the RFC 1483 bridged encapsulation really uses the 8-byte header (there are numerous options in the RFC).

see 10 comments

Line rate and bit rate

Rajendra had an interesting problem:

Recently I got confused with the term line-rate. Is it the packets being switched across the switch fabric or control packets destined to the protocol tasks or both or something else?

The line rate is a physical layer term that has nothing to do with the line cards or switching fabrics. It indicates the actual speed with which the bits are sent onto the wire (and is thus also known as physical layer gross bit rate). The data transfer rate (commonly known as bit rate) is the transfer rate offered by the physical layer to the data link layer. If you want to be precise, you should call it physical layer net bit rate.

Two well-known physical layer technologies with different line rate and data transfer rate are ISDN (actually the I.430 recommendation) with 160192 kbps line rate and 144 kbps data transfer rate and Gigabit Ethernet (the 802.3z recommendation) with 1.25 Gbps line rate (due to 8b/10b encoding).

This article is part of You've asked for it series.

see 3 comments

Bandwidth allocation with class-based weighted fair queuing (CB-WFQ)

Sebastian sent me an interesting question:

I have read that we can only use 75% of the bandwidth for the custom queues as 25% is reserved for the keepalives and routing protocol updates. If I want to set 50% of bandwidth for a particular queue should it 50% of the total bandwidth or 50% of the available 75% of the bandwidth?

Before going into the details, it’s important to remember that the WFQ (CB-WFQ is only a mechanism to sort packets into output queues) uses relative ratios (percentages) between queues to determine which packet to send (the absolute bandwidths are used just to compute the ratios).

To prevent the classified traffic from starving the routing protocols and other important packets (ARP, CDP, ILMI, Frame Relay keepalives …), the router does not allow the class queues to consume more than 75% of the total interface bandwidth. The remaining 25% are used for the default class (all non-classified traffic) as well as all non-IP packets. You can change the limit with the max-reserved-bandwidth interface configuration command; just make sure you have plenty of bandwidth left for the control traffic.

When you configure the class-based queuing with the bandwidth [remaining] percent percent command, the number you specify is the percentage of the total interface bandwidth and is used to determine the ratios in the WFQ system. Obviously, the sum of all percentages and the max-reserved-bandwidth should not be more than 100.

When using the bandwidth kbps command, the traffic in the class you’re configuring will get at least the specified amount of bandwidth. The bandwidth percentage is computed as the specified bandwidth divided by the interface bandwidth.

The interface bandwidth can be set based on the actual hardware (it’s not hard to guess what the bandwidth of a 100BASE-TX interface is), computed for multilink bundles or configured with the bandwidth interface configuration command.

This article is part of You've asked for it series.

see 7 comments

Time-based IOS actions

One of my regular readers asked this question:

I want to shut my particular interface after 6:oo pm every day. Right now I have just the general idea as to how EEM works. What I guessed is that when the router's clock ticks 6:00 pm a syslog message might be generated and event manager will catch it and execute the particular command. But I don’t understand how EEM monitors the clock so that it knows its 6:00 pm now and it has to generate a syslog message.

If you want a quick answer, you’ll find the exact solution in the Time-based wireless interface activity article in the EEM category in the CT3 wiki. However, you might want to get a bit more out of this question.

If you want to use EEM, the best investment of your time would be to spend an hour or two reading its introductory documentation, particularly the section describing the event detectors. It nicely describes four time-based event detectors available in EEM. You can also try searching my blog and CT3 wiki. For example, I’ve written about EEM timer cron event more than two years ago. The blog labels and wiki categorization should be of great help: start with the EEM label in the blog and the EEM category in the CT3 wiki.

Last but not least, there’s always more than one way to solve a problem with Cisco IOS. If you’re not happy with EEM, you can use Kron.

This article is part of You've asked for it series.

see 3 comments

Over a million page views

Two weeks ago Douglas Crockford wrote that his blog generated more than a million page views. Since he's an icon in the mainstream Web industry with his own Wikipedia listing, I simply had to check how my blog compares to that. Not bad, claims I had 1.2 million page loads in two and a half years.
If you'll remain as loyal readers as you've been in the first three months of this year, we'll probably reach the million-a-year mark in 2009.
see 4 comments

Blocking rogue DHCP servers

The reader who was concerned about making a loop while connecting a switch to itself was also facing “customer-installed” DHCP servers in his LAN. He wrote:

Some users have installed their own Linksys routers and plug our cable in router's LAN ports, so there is DHCP servers fight in our LAN. How can I sort this out (I cannot physically find the location of the Linsys routers)?

The ideal solution is DHCP snooping (assuming your switch supports it), well documented on The basic configuration takes only a few minutes:

  • Enable the feature globally with the ip dhcp snooping global configuration command.
  • Enable the feature for individual VLANs with the ip dhcp snooping vlan number global configuration command.
  • Configure the trusted interfaces with the ip dhcp snooping trust interface configuration command.
  • Rate-limit DHCP on untrusted interfaces with the ip dhcp snooping limit rate interface configuration command.

This article is part of You've asked for it series.

see 7 comments

BGP Local-AS feature: the basics

I’ve always thought that Cisco introduced the BGP Local-AS feature into IOS to support complex MPLS VPN design scenarios. Obviously I was wrong, the early documentation always describes an ISP AS merging scenario. Unfortunately, all the articles I’ve found skip some important details: they describe the basics and the configuration commands, but forget to mention the impact on the AS paths received by the ISP customers. Obviously we need a more thorough description of this feature.

Read the whole article in the CT3 wiki

see 2 comments

Last-resort password recovery

Pappyar has sent me an interesting password recovery technique, which can be used in those weird circumstances when you cannot force the router to go to ROMMON (for example, you’ve configured no service password-recovery and the break signal does not work as expected). Unfortunately, his trick works only if you can remove the flash memory from the router (it’s soldered in low-end models):

  1. Turn off the router.
  2. Take out the flash.
  3. Turn on the router.
  4. This time router will take you to ROMMON as it cannot find an IOS image.
  5. Set the configuration register with confreg 0x2142.
  6. Reset (this will change the stored value of the configuration register).
  7. Turn off the router.
  8. Reinsert the flash.
  9. Turn on the router and you are done.
see 10 comments

Which is the “best” PE-CE routing protocol?

A comment to my post “Video:Small remote site using BGP as PE-CE routing protocol” has made me think very hard about the PE-CE routing protocol selection. Justin was right: although it’s best for the Service Provider to try to push BGP as far out as possible, it might make customer’s life more complex in some scenarios. I’ve decided to write an article about these PE-CE routing protocol issues, but as I’ve started writing it, it dawned on me: this is not a technical issue, but a business one. You can find my reasoning in the “Choosing customer MPLS VPN routing protocols” article published by

The list of all articles I wrote for SearchTelecom is available in the CT3 wiki.

see 6 comments

Off-topic: Search engine wars are over

StatCounter, the free web tracking service I'm using has recently launched global statistics tool that allows you to analyze the statistics of the data they're collecting from thousands of participating web sites. An interesting one is the "Search engine" graph:
As you can see, Google has approximately 90% market share, Yahoo (the second place) has less than 10%, the others are clearly irrelevant.
see 1 comments

Recovering from expired one-time username

A reader sent me an interesting question:

Do you have any advice for resetting/logging into a router (2821) where the one time user of cisco:cisco has already been used?

I couldn't offer any better advice than performing the regular password recovery procedure. Is there another solution?

This article is part of You've asked for it series.

see 4 comments

EBGP load balancing with a multihop EBGP session

If someone asks you to implement load balancing on equal-bandwidth parallel links between a pair of BGP routers, your first thought should be multihop EBGP session between loopback interfaces. Although this design is ancient and excruciatingly boring, it still has a few interesting twists, for example:

Last but not least, an EBGP multihop session is vulnerable. You can use the neighbor disable-connected-check command to have a single-hop EBGP session with an IP address that is not directly connected.

Read the whole article in the CT3 wiki

see 4 comments

Missing routes when running IS-IS over Frame Relay

Fat finger follow-up: use the key labels

Karsten Iwen made an interesting comment to my “Don't let a lab rat anywhere near a production box” post: you should avoid the SSH/VPN key generation mistakes by using key labels. He also wrote a post explaining the concept but since it’s in German, let me rephrase it in English.

Cisco IOS release 12.2(8)T added the label parameter to the crypto key generate rsa command. You can use this parameter to assign a label to your VPN key, for example

Rtr(config)#crypto key generate rsa label VPN modulus 2048

To use the labeled key to generate your certificate, use the rsakeypair command in the CA-trustpoint configuration mode:

crypto pki trustpoint
 enrollment retry count 100
 enrollment mode ra
 enrollment url
 rsakeypair VPN
see 1 comments

Build a VPN across your IP network with Multi-VRF feature

One of our customers had to provide end-to-end IP transport across their enterprise network for an outsourced video surveillance solution. We implemented a true VPN solution for them (the hosts in the enterprise network cannot access the surveillance equipment and vice versa) using the Multi-VRF feature available in all recent Cisco IOS releases. The solution is described in the March IP Corner article “Add a VPN to an Enterprise Network with Multi-VRF Functionality”.

Add comment

Don't let a lab rat anywhere near a production box

I tried to do a few simple NETCONF tests yesterday (I wanted to see how the router's configuration looks like when it's encoded in XML). I didn't want to start a lab for such a simple task and decided to use my home router. SSH was the only reasonable transport (you can't run BEEP with standard Linux tools), but it was not working on the router.

Obviously I've configured too many lab devices in the past. As soon as I've realized I had SSH problems, my fingers automatically typed crypto key generate rsa. A few milliseconds after I've hit ENTER it dawned on me that my router uses PKI certificates for the VPN connection to our network ... and I've managed to invalidate the router's certificate, which is one of the few things that a reload will not solve. Although our IT guru was fantastic and approved my (router's) certificate request late in the evening, I still felt bad about the whole experience.

And the moral? Don't let a lab wizard with fast fingers too close to a production box :)

see 1 comments

Generating syslog messages from Tcl

If you use Tcl to write Embedded Event Manager policies, you could use the action_syslog command to generate syslog messages. In all other Tcl-based environments (including tclsh), this API is not available, but you could use the syslog: file system to generate debugging messages.

Read more in the CT3 wiki article Generating Syslog messages from Tclsh.

The article is part of Tclsh on Cisco IOS tutorial.

see 1 comments

Four byte AS number support in Cisco IOS

Last week IOS release 12.4(24)T appeared on CCO. One of the significant improvements in that release (I can’t manage to get enthusiastic about new kludges to support the SIP morass) is the support for 4-byte BGP AS numbers.

Finally an enterprise network that uses Cisco routers to connect to the Internet can use the new AS numbers distributed by the regional registries (assuming you’re brave enough to run 12.4(24)T on your production gear). The Service Providers using 7600 routers will have to wait … corresponding 12.2SR release is not yet available.

Hat tip to Tassos @ CCIE in 3 months: he was the first to write about this feature.

see 2 comments

AS-path prepending: technical details

I thought I knew all there is to know about the AS-path prepending before last month’s incident, which prompted me to focus on this particular Cisco IOS feature.

For example, did you know you could do inbound AS-path prepending? I didn’t, until Rodney Dunn from Cisco mentioned it in an e-mail exchange. Did you ever wonder whether the AS-path prepending affects inbound or outbound AS-path filters? I had a hunch it doesn’t, but was never sure.

You can find everything you never wanted to know about AS-path prepending in the AS-path prepending (technical details) article in the CT3 wiki.

Add comment

AS-path incident: the bowdlerizer strikes

A post by CiscoSubnet has pointed me to the official IntelliShield alert describing the AS-path incident. While the alert is very well written (I wouldn’t expect less) and the associated Protecting Border Gateway Protocol for the Enterprise is excellent, I was highly amused by the following text:

Cisco Security Intelligence Operations has identified a method through which administrators can modify device configurations to mitigate the effects of the AS path processing issue. Administrators can limit the amount of AS path segments that are associated with any route by using the bgp maxas-limit feature …

The bgp maxas-limit command was suggested by some members of the NANOG mailing list as early as few hours after the incident (which happened on February 16th; the IntelliShield alert has been released on February 20th) and I’ve released a detailed article on February 17th. I know Cisco’s engineers did a great job in this particular case; why did someone have to run their results through the /usr/local/marketese | /usr/bin/bowdlerize?

see 1 comments