Building Network Automation Solutions
6 week online course starting in September 2017

Hierarchical Queueing Framework: queue limits and output drops

The QoS behavior in Cisco IOS has changed significantly with the introduction of the Hierarchical QoS Framework (HQF) in IOS release 12.4(20)T. Cisco is slowly producing in-depth articles describing the changes; the first one I’ve found documents the old and new output queue limits and output drops.

I’ve also added the link to this article to the Further reading section of my Queuing principles in Cisco IOS article.

IPv6 labs are no longer available on Cisco’s Partner Education Connection

Csaba asked me about the availability of our IPv6 remote labs on Partner Education Connection. He wrote:

I was happy when I found this information, but after a while I realized that IPv6 labs are no more available on PEC site. Do you know any details why this topic has been removed from PEC or it can be that IPv6 is still there just I couldn't find it?

Cisco has removed almost all external remote labs from PEC in October. To increase the confusion, some of them might still appear in the catalog, but you cannot start them. The only thing we could do was to inform the users landing on our web site that the labs are no longer available and that they could buy them from our product catalog.

IPv6-capable or IPv6-ready: is it enough?

During the IPv6 summit in Slovenia I’ve participated in a roundtable organized by our Ministry of Higher Education, Science and Technology. One of my points was that the government should require true IPv6 support in all its IT procurement processes to promote IPv6 adoption (I have to admit I’ve borrowed a few ideas from Geoff Huston’s “Is the Transition to IPv6 a Market Failure?” article) … and one of the participants (coming from the Service Provider industry) answered that “that’s common hygiene”. I’m not so sure.

Topics like this are covered in my Enterprise IPv6 Deployment workshop. Learn more about my workshops from my web site.

Followup: What’s wrong with the Zone-Based Firewalls book

I’d like to thank all the readers that took time and responded to my question about the failure of my Deploying Zone-Based Firewalls book. The sad short conclusion is: while everyone would love to have an electronic copy of the book, the technology and the mindsets are simply not ready yet. Here are the details:

What’s your take on Alcatel-Lucent IP+Optical integration?

Approximately a month ago Alcatel-Lucent launched Converged Backbone Transformation (are they sharing marketing wizards with Cisco … or is the excessive hype an industry-wide phenomenon?): a visionary(?) convergence of IP and optical technologies. If you haven’t heard about it yet, you could try to start with the IDC report published on Alcatel-Lucent’s web site (I’m always amazed how some people manage to tell so little in so many words).

Once you get past the fluff to the details, it could be that they're implementing a lot of common-sense. For example, it looks like the lambda-level grooming replaces GBIC/SFP transceivers with something that can generate multiple lambdas on the router and feed these lambdas directly into the DWDM gear. In my understanding, it replaces the GE port-GBIC-fiber-GBIC-GE port-lambda generation-DWDM chain with the shorter and cheaper GE port-lambda GBIC-fiber-lambda port-DWDM chain (obviously, I might be completely wrong; it’s hard to deduce the details from a press release).

Anyhow, I’d really appreciate your thoughts on this launch. Does it make sense? How does it compare to what Cisco and Juniper are doing? Is this a move in the right direction … or is Alcatel-Lucent playing a catch-up and trying to cover it with a grand marketecture?

IOS packaging: Moore’s Law Won

Great news: Cisco launched a new series of midrange routers on Tuesday. They're very probably great products (I wouldn't expect less from Cisco). Also as expected, their marketing department couldn’t help itself (yet again) and had to position the launch as a universe-changing event: this time they Revealed the Borderless Network and spent loads of money producing “viral videos”. OK, maybe their average customer is stupid enough to fall for those tricks; I’m positive you’re not … so let’s see what’s really new (here's what Cisco admits is new after you've got past all the marketing fluff):

Report interface loss based on OSPF neighbor loss

Nicolas sent me an interesting problem: he has numerous point-to-point GRE-over-IPSec tunnels on his core router and detects remote site failure with OSPF neighbor loss events. He would like to receive an e-mail when an OSPF neighbor goes down (quite easy to do with EEM), but would also like to receive interface description in the e-mail subject to simplify his troubleshooting.

With the regular expressions available in EEM 3.0 you can extract interface name from syslog message, execute show interface command and extract the interface description from it.

The EEM applet source code is available in the CT3 wiki

This article is part of You've asked for it series.

Help appreciated: what’s wrong with my Zone-based Firewalls book?

A quick question for you: in two years since my Deploying Zone-based Firewalls digital short cut (marketese for downloadable PDF) was published, we’ve sold around 200 copies of it. Obviously we’re doing something wrong and I’d appreciate your opinion: is it the topic (are you using ZB firewall on Cisco IOS?), the format (would you prefer paper copy?), the platform (Cisco IOS as a firewall), pricing ($14.99 for 112 pages) or something else?

What went wrong: TCP lives in the dial-up world

As expected, my “the socket API is broken” post generated numerous comments, many of them missing the point (for example, someone scolded me for quoting Wikipedia and not the official Linux documentation). I did not want to discuss the intricate technical details of the various incarnations of the API but the generic stupidity of having to deal with low-level networking details in the application.

Fabio was kind enough to provide the recommended method of using the Socket API from man getaddrinfo, effectively proving my point: why should every application use a convoluted function when all we want to do (in most cases) is connect to the server.

Patryk went even further and claimed that the socket API provides “basic functionality” and that libc is not the right place for anything more. Well, that mentality caused most of the IPv4-to-IPv6 application-related issues: obviously the applications developed before IPv6 was a serious consideration had to be rewritten because all the low-level code was embedded in the applications, not isolated in the library. A similar problem has effectively stalled SCTP deployment.

However, these are not the only problems we’re facing today. Even if the application properly implements the “try connecting to multiple addresses returned by DNS” function, the response time becomes unacceptable due to the default TCP timeout values coded in various operating systems’ TCP stacks.

For example, it takes up to three minutes for a TCP connect call to timeout on a Fedora-11 Linux distribution (the connect call aborts immediately if an intermediate router sends back an ICMP unreachable reply and the ARP timeout causes an abort in three seconds). Windows XP is slightly better; the default timeout is set at 20 seconds.

You might wonder what prompted the TCP designers to choose these exceedingly large values. TCP was designed more than 20 years ago when the analog dialup modems were commonly used to connect to the Internet. These modems could take a minute (or longer) to establish the connection and if you wanted to have a reliable TCP session setup, you had to wait significantly longer before aborting the session setup system call. The Internet has changed dramatically in the meantime, but nobody ever bothered changing the defaults.

If you want to rush and write a comment how the default can be changed, you’re yet again missing the point: we cannot implement multihomed IP hosts using more than one IP address due to the crazy default TCP timeout values. As soon as the first address becomes unreachable, the session establishment time (for an average user using out-of-box software) becomes unacceptable.

IPv6 summit in Slovenia

The Slovenian IPv6 summit where I presented my views on IPv6 deployment in enterprise networks was a huge success, with over 130 attendees (not bad for a country with only 2M people) and Martin Levy from Hurricane Electric as the keynote speaker. The event (the program as translated by Google) was organized by fantastic people from the Go6 Institute. Unfortunately the go6 web site is only in Slovenian (try Google translator if you want to be highly amused or totally confused), but you can view my presentation, read the Hurricane Electric press release or browse the photos (I’m in the right-hand picture in the third row).

Someone took notes and posted them on the go6 web site. The notes for Martin’s keynote and my presentation are in English (just search for our names).

Last but not least, if you’re based in Europe and you’d like me to deliver my presentation at your local IPv6 event, send me a message.

Fantastic DDoS protection: it’s getting worse

Last week I described the “beauty” I’d discovered through the NetworkWorld site: a solution that supposedly rejects DoS frames in 6 nanoseconds. Without having more details, I’ve tried hard to be objective and justify that you cannot get that performance in a best-case scenario (at least without having really expensive hardware and optimized architecture). In the meantime, one of the readers provided the name of the author of this discovery and I was able to find the original publication that was published in the Proceedings of the 2007 spring simulation multiconference by Society for Computer Simulation International.

Follow-up: Interface default route

Judging by your comments, some of you have already faced a stupidity similar to the one I’ve described on Friday (BTW, I’ve remembered this particular debacle when receiving a Pingsta case invitation with very similar symptoms). The symptoms are well described in the comments: the CPU utilization of the ARP process increases, packet forwarding becomes sluggish and the router runs out of memory, potentially resulting in a router crash. Now let’s analyze what’s going on.

My stupid moments: Interface default route

Years ago I was faced with an interesting challenge: an Internet customer was connected to our PE router with an Ethernet link and I did not want to include the PE router’s IP address in the default route on the CE router.

The latest IOS release in those days was probably somewhere around 11.x; none of the DHCP goodies were available.

After pondering the problem for a while, I got a brilliant idea: if I would use an interface default route, proxy-ARP would solve all my problems. This is the configuration I’ve deployed on the CE-router:

interface Ethernet 0
 description Uplink to the ISP
 ip address 10.0.1.2 255.255.255.0
!
ip route 0.0.0.0 0.0.0.0 ethernet 0

We tested this configuration in the middle of the night and it worked as expected. What do you think happened in the morning?

DHCP client address change detector

In a previous post I’ve described how useless DHCP logging is when you try to detect change in DHCP-assigned IP address. Fortunately the removal of the old IP address (triggered by the DHCPNAK server response) and configuration of the new IP address (sent in the DHCPACK response) triggers a change in the IP routing table that can be detected with the new IP routing table event detector introduced in EEM 3.0 (available from Cisco IOS release 12.4(22)T).

The EEM applet that detects the changed DHCP-assigned IP address (including the initial address assignment which also triggers the syslog message) is described in the CT3 wiki.

Will they ever start using their brains?

This morning I’ve discovered yet another journalistic gem. It started innocently enough: someone has announced prototype security software that blocks DDoS attacks. The fundamental idea (as explained in the article) sounds mushy: they’ve started with one-time user ID and introduced extra fields in the data packets. How can that ever scale in public deployment (which is where you’d be most concerned about a DDoS attack)?

But the true “revelation” came at the beginning of page 2: this software can filter bogus packets in 6 nanoseconds on a Pentium-class processor. Now let’s try to put this in perspective.

SSH RSA authentication works in IOS release 15.0M

The feature we’ve begged, prayed, sobbed, yelled, screamed for has finally been implemented in Cisco IOS: public key SSH authentication works in IOS release 15.0M (and is surprisingly easy to use).

After configuring SSH server on IOS (see also comments to this post), you have to configure the ssh pubkey-chain, where you can enter the key string (from your SSH public key file) or the key’s hash (which is displayed by the ssh-keygen command).

It’s probably easier to copy/paste the public key from your id_rsa.pub file into the terminal window …

R2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R2(config)#ip ssh pubkey-chain
R2(conf-ssh-pubkey)#username pipi
R2(conf-ssh-pubkey-user)#key-string
R2(conf-ssh-pubkey-data)#$AAQEA6jYlf9MBskhkWov+ZOUDKun0ExQIRj1zfWA/YciO02VS  
R2(conf-ssh-pubkey-data)#$XsxM7SqNkRSQOR7y7HBMoxTHV7o+R/uS6A8/mF0A3P/ScRjct  
R2(conf-ssh-pubkey-data)#$JrNGACGaFy1njD9PrrvrU4o4hx6XDr6xVXF4sP4OCSXIn+Cp8  
R2(conf-ssh-pubkey-data)#$bCnZLmv908AeDb1Ac4nPdsn1OhCPIg6fxZjB7DvAMB8Dbr+7Y  
R2(conf-ssh-pubkey-data)#$apEbGE94luIqnBc61HsMd6JCWbQ== pipi@fedi.nil.si     
R2(conf-ssh-pubkey-data)#exit
R2(conf-ssh-pubkey-user)#^Z

… and let the router convert it into the key hash, which is stored in the configuration:

R2#show run | section ssh
ip ssh rsa keypair-name SSH
ip ssh version 2
ip ssh pubkey-chain
 username pipi
  key-hash ssh-rsa C20B739F2645D6850C591C6A11780CB5 pipi@fedi.nil.si

After this simple step, you can log into your router without typing the password. Finally we have a manageable way of secure remote command execution.

IOS release 15.0

This is not an April 1st post: I’ve just realized that Cisco quietly released IOS 15.0M (mainstream). Haven’t tested it yet, but the images for a large variety of platforms are already available on CCO. The new features listed in the documentation include:

  • Full BFD support, including static routes, BFD-in-VRF and BFD-over-Frame Relay (next step: test it on a 2800-series router);
  • DHCP authentication;
  • DMVPN tunnel health monitoring;
  • EEM 3.1 (whatever that is, the EEM documentation hasn’t been updated yet);
  • Interaction between IS-IS and LDP;
  • BGP local convergence in MPLS VPN networks (the feature has already been available in 12.2 SRC, now it’s available on more platforms);
  • OSPF graceful shutdown and OSPF TTL security check features are available on more platforms;
  • Intra-zone traffic inspection in zone-based firewall.

It looks like (as expected) the 15.0 release is a grand merge of all previous IOS trains (with a few extra features). Good job; finally we have something new to play with :)

IOS fossils: Classful OSPF redistribution

In the classful days of the Internet it made sense to limit the amount of information redistributed between the routing protocols. OSPF was always classless, but RIPv1 wasn’t … and you could get all sorts of crazy routes from RIP that would mess up the rest of your network if they would ever get redistributed into OSPF. To prevent that, Cisco’s engineers introduced the subnets option in the OSPF redistribute command.

Either the OSPF redistribute command is really old (before the distribute-list command started accepting extended ACL which could filter on the subnet mask) or someone was too dumb to use the extended ACL and Cisco had to provide an obvious workaround.

By the time Cisco implemented EIGRP and BGPv4 (IOS release 9.21, 15+ years ago), the absurdity of the classful redistribution was already obvious. These routing protocols accept whatever routes you want to redistribute and their variants of the redistribute command don’t have the subnets keyword. However, nobody ever took steps to remove this fossil from the IOS code.

I wouldn’t care if we would be talking about an obscure option like the OSPF-to-BGP redistribution tags, but the OSPF redistribute command is one of the most confusingly harmful IOS routing protocol commands (the only one getting close in my opinion is the auto-summary in EIGRP). One of the IOS releases introduced a warning that would be printed whenever you’re configuring redistribution into OSPF without the subnets keyword (great job: wouldn’t it be better to fix the problem?) …

rtr(config)#router ospf 999
rtr(config-router)#redistribute static
% Only classful networks will be redistributed

… but the warning is incorrect. Yuri Selivanov sent me an interesting observation: while the subnets don’t get redistributed without the subnets keyword, supernets do. The “classful” filter is not even working correctly (or, at the very least, it’s not doing what the IOS claims).

I am positive Cisco will never fix this problem and we’ll have to cope with this command until the last bits of IPv4 code are erased from Cisco IOS, but here are a few things they could have done:

  • Remove the classful redistribution filter from the code and make the subnets keyword obsolete. I sincerely doubt any reasonably-sized network is running classful redistribution with recent IOS code (and who cares about the network cores running on AGS+).
  • Add another option to the OSPF process that disables the filter. This option would have to be explicitly stored in the router configuration like the log-adjacency-changes OSPF configuration command (to ensure the old configurations work) but should be turned on for all new instances of OSPF routing protocols.

Undoubtedly a few people (mostly old-timers) would get confused once or twice, but the number of OSPF support cases might be “slightly” reduced.