Building network automation solutions

9 module online course

Start now!

Blog Posts in February 2008

Building customer-resilient BGP networks

When Kate Gerwig, my wonderful editor from, and myself agreed on the contents of the “Building customer-resilient BGP networks” article, we had no idea that it would become extremely relevant just days before it was published. The article describes the tools a Service Provider should use to ensure that its customers cannot harm its BGP routing data (and consequently its other customers and the Internet at large).

On February 24th, someone in Pakistan decided to block local access to YouTube … and someone else decided that the best way to approach the problem was to block the whole world’s access to YouTube.

YouTube (AS 36561) advertises its networks as nicely aggregated IP prefixes (one /20, one /22, one /19 and one /24). The hijackers decided to block the /22 range by announcing more specific (/24) prefixes. Their upstream provider had no filters in place and accepted these prefixes (BIG mistake). They soon propagated throughout the Internet, until someone in YouTube noticed them and started originating the same /24 prefixes.

The timing and sequence of events is outlined in the Renesys blog. Full in-depth analysis including an animation of the BGP changes is posted on the RIPE NCC web site.

Since YouTube is way better connected to the major Internet providers as the offenders, from most points in the Internet the AS paths of the YouTube-originated prefixes were shorter than the ones the hijackers used, resulting in corrected routing. Ten minutes later, YouTube started announcing even more specific prefixes (/25) that completely restored connectivity to their servers.

I’m just guessing that someone had to remove the usual filters that drop the /25 (and longer) prefixes from being accepted from the customers in the meantime (Good work!).

The big debate people are having these days is: “can anything be done to stop this”. Of course there is a solution: major Service Providers use Internet Routing Registries and document their autonomous systems, routes and routing policies (and every AS should be doing the same). However, the solution is not enforced globally and while some Service Providers use the registries to build filters for their customers, no-one is willing to pull the plug and implement the filtering policies on all ingress/egress points.

Unfortunately, you cannot implement the IP prefix filters on major exchange points, as there’s no reasonable way you could download the prefix lists covering the full Internet routing table into the routers and filter hundreds of thousands of IP prefixes. The last chance to implement the filters is on the regional level.

To get a better perspective of the problem, compare the entry for AS3491 (PCCW Global, the upstream provider that propagated the fake Pakistani route) with the entries for AS 3356 (Level 3) and AS 3561 (Savvis) … and allow me to refrain from commenting what you see.

The list of all articles I wrote for SearchTelecom is available in the CT3 wiki.

see 1 comments

Unconditional trunking port on a Catalyst 3560

Rob van Ooijen has sent me a really interesting question:

I've configured a switch port to be unconditionally a trunk with the switchport mode trunk configuration command. However, when the interface was enabled, I've got a dynamic trap indicating the trunking was still dynamic (and the show commands also showed negotiation of trunking is enabled).

In fact, adjacent layer-2 devices can negotiate a number of things these days, among them:

  • Speed and duplex settings (we'll leave this can of worms safely closed);
  • Trunking (a port can be access or trunk port);
  • Trunk encapsulation (ISL or 802.1q);

With the switchport mode trunk command you just disable the trunking negotiation (the port will never become an access port), but the switch still runs Dynamic Trunking Protocol (DTP) on it, unless you disable DTP with switchport nonegotiate, which (in combination with all the other configuration commands) makes the port configuration totally static. More configuration can be found in Cisco's documentation and in the Basics of DTP document.

This article is part of You've asked for it series.

add comment

Merging VTY configurations

Someone has sent me an interesting question a while ago: he's changed the configuration of a single VTY line and got three blocks of VTY configuration commands, similar to this:
line vty 0 2
line vty 3
 password secret
line vty 4
He wanted to merge the three configuration blocks back into a single one but somehow didn't know how to do it.

To realize what's going on, you have to understand how the IOS generates line configurations. It takes the first line (VTY 0, for example) and generates its configuration. If the next line (VTY 1) has exactly the same configuration, the range of numbers is expanded (becoming VTY 0 1) and so forth until the pool of similar lines is exhausted or a line is found that has at least one parameter different from the starting one, in which case a new block is started. That's why the sample configuration has three blocks (0-2, 3 and 4) even though the first and the third block are identical.

However, if you change the offending parameter, the VTY lines will have identical configurations and will be automatically merged. If you want to be on the safe side, you should change the parameter for all lines, for example:
line vty 0 4
 password secret

Note: This article is part of You've asked for it series.

see 2 comments

Reduce IP addressing errors in lab environment

One of the most tedious tasks in the initial lab setup (at least for me) is the IP address configuration, which usually includes a number of typos and mixups on the WAN links. You can simplify then WAN address configuration if you configure only one end of the WAN link and let PPP do the rest. For example, you could use the following configuration to configure WAN link on your core router …
hostname Core-2
interface Serial1/0
 description link to POP
 ip address
 encapsulation ppp
 peer default ip address
… and use IPCP negotiation on the POP router to pick up the WAN IP address:
hostname POP
interface Serial1/0
 description link to Core-1
 ip address negotiated
 encapsulation ppp

You should not configure no peer neighbor-route on the router that gets dynamic IP address, as the subnet mask is not assigned with IPCP; you need the IPCP-generated host routes if you want to do hop-by-hop telnet between the routers.

see 1 comments

Time-based BGP policy routing

Petr Lapukhov describes an interesting scenarion in his post BGP Time-Based Policy Routing: a multi-homed customer that uses one upstream link (for example, more reliable but slower one) during the work hours, switching to the other upstream link (faster, less reliable) after that.

He uses BGP communities to achieve the switch (perfect solution if your ISP supports them) and time-based ACL in a route-map to set the community based on time-of-day. As Cisco changed the way BGP imports local routes in IOS release 12.3T, he then devises an ingenious solution based on reliable static routing to trigger a change in the IP routing table.

The optimum solution is way simpler: you just configure two EEM applets to perform clear ip route network command at appropriate times.
read more see 2 comments

Running OSPF across Frame Relay

The various models you can use to run OSPF over partially-meshed WAN (usually Frame Relay) are complex enough to make you scream ... or switch to EIGRP :) Arden Packeer did an excellent job in his three-part tutorial describing broadcast and non-broadcast network types, point-to-multipoint network type and point-to-point links over Frame Relay.

If you still have problems with OSPF after reading all three tutorials, use point-to-point subinterfaces.

add comment

Intrusion Prevention System (IPS) remote labs are available free-of-charge to Cisco partners

We've recently deployed remote labs associated with the Implementing Cisco Intrusion Prevention System v6.0 on Partner Education Connection; they are thus available free-of-charge to all Cisco partners. The following exercises are available:

If you're a Cisco partner, you can start any of the listed lab exercises simply by clicking on its name and supplying your CCO username/password when asked for it by Cisco authentication web page.

Everyone else can buy the same labs from our learning store. For example, for €120, you can get a one week unlimited access to the labs with the ability to repeat every exercise as ofter as you need.

add comment

Building Core Networks with BGP, OSPF and MPLS

Do you need advanced knowledge and skills needed in designing and implementing core MPLS networks? We have developed exactly the course you need.

The Building Core Networks with BGP, OSPF and MPLS (NIL_BCMPL) course focuses on MPLS applications such as MPLS VPN (with special attention to Internet access from a VPN), Any Transport over MPLS (AToM), Carrier Supporting Carrier (CsC) and MPLS Traffic Engineering (MPLS TE). As a prerequisite for MPLS deployment the routing protocols are explained as well - the Open Shortest Path First (OSPF) and Border Gateway Protocol (BGP). The scalability issues of the protocols as well as multiprotocol BGP are addressed as well. Several practice labs enable you to gain the necessary experience in deploying the MPLS-based core networks.
see 1 comments

“Cisco IOS Hints” blog featured in NetworkWorld

Network World has published an article describing 20 useful sites for Cisco networking professionals, featuring among other sites the Cisco IOS Hints blog. I felt even better when I've discovered my blog was one of the three non-US sites mentioned in the article :)

It's well worth browsing through the list. I already had most of these sites in my feed reader and I've discovered a few new nuggets.
see 3 comments

Use all the tools you have

BGP implementation on Cisco IOS gives you a number of filtering options, including prefix filters, AS path filters and route maps. While it might be tempting to learn just the most versatile tool available (route maps) and discard all the others, judicious use of all available tools can simplify your router configurations.

For example, an Internet Service Provider might want to filter incoming updates received from the customers to ensure they’re not advertising transit routes and advertise only IP prefixes they actually own. Inbound route maps might also be needed to attach BGP communities to inbound routes or set BGP attributes (for example, local preference) based on communities attached to incoming routing updates.

You can perform all these tasks with route-maps, but then you’d probably have to create a dedicated route-map for each customer (as the inbound prefix filter has to be customer-specific). Changing your routing policies or community definitions would require changing a lot of route maps.

Update [email protected]:39: distribute lists and prefix lists can't coexist (they cannot be configured in the same direction on the same neighbor)

On the other hand, if you use all the filters available in the BGP routing process, you could:
  • Use neighbor filter-list in to check the customer-specific AS path requirements;
  • Use neighbor prefix-list in to filter customer prefixes, reject too long prefixes or prefixes not belonging to the customer;
  • Use neighbor route-map in to drop prefixes belonging to your own address space, implement various routing policies and set BGP communities.

The solution scales even better if you configure common filters (route-map in and filter-list in in our scenario) in a BGP peer session template.

see 2 comments

Differentiating between port scanners and legitimate users

One of my readers asked a very interesting question:
“Is there a way to have a port on a router open for legitimate use while closed to port scanning software and the such. For example. I have SSL VPN configured on my IOS router. Is it possible to have the port seem stealthed to port scanners while still allowing legitimate access to the service? An example being, allowing a web browser to connect using the port but making sure that a port scanner doesn't detect it as open.”
The short answer is no, unless you can differentiate legitimate users by their IP addresses. The port scanners (when using SYN scan) simply open and close a TCP session, and there is no way for a router to differentiate between the legitimate users (who would send valid HTTP GET requests) and port scanners (that would close the session as soon as it's established).

If you can distinguish between legitimate users and everyone else based on their IP address, the task becomes simpler: either you apply inbound access list on Internet-facing interface of the router or configure per-service access-list (for example, ip http access-class acl). When using the inbound access-list, the port appears filtered or stealth, when configuring per-service access-class, it's reported as closed.
add comment

Remove unwanted PPP peer route

When IOS started supporting dynamic allocation of IPCP (IP over PPP) addresses, it also got the mechanism to insert a dynamic host route toward the neighbor's IP address once the IP addresses were negotiated. This mechanism makes perfect sense in dynamic address allocation environments, as the subnet mask is not negotiated with IPCP. Without a host route toward the other end of the PPP link, there would be no easy way to reach the IP prefixes reachable via the PPP link.
read more see 1 comments

Implement “wc -l” in Cisco IOS

Sometimes it would be nice to have the full complement of Unix utilities available on Cisco IOS. That's not going to happen for a while, but we can use Tcl to make our life simpler in the meantime. Xavier Brouckaert, a regular contributor to my blog, has sent me the Tcl implementation of line counting utility (equivalent to wc -l on Unix).

First you have to define the wc Tcl procedure:
proc wc { cmd } { llength [split [exec $cmd] "\n" ] }
You can define the procedure interactively in Tclsh (but then you have to do it every time you start Tclsh) or you could store the code in a flash file and execute the file every time the Tclsh is started with the scripting tcl init filename global configuration command.

Once the wc procedure is defined, execute wc { IOS command } in Tclsh and you'll get the line count. For example, to get the number of directly connected routes use
wc { show ip route ¦ include ^C }

The include ^C filter includes all lines that start with letter C; in our case all directly connected routes

Obviously you could turn this idea into a full-blown Tclsh script that would accept CLI arguments … but I'll leave this as an exercise for the readers (you can probably tell I've been reading some academic literature lately :). However, if you find the time to write a more complete wc implementation on IOS, please do post the URL here.

see 2 comments

IPv6 e-learning solution

Do you want to gain IPv6 configuration skills and test the associated routing protocols at the time that suits you most? The IPv6 e-course allows you to do just that.

The IPv6 Fundamentals, Design and Deployment (IP6FD) e-course is a blended learning solution that consists of the IP6FD web-based training and associated remote lab bundle. The course provides you with knowledge and skills needed for transitioning to IPv6 based networks. The content encompasses design and security considerations, IPv6 configuration principles and IPv6 transition mechanisms. You will learn how to implement IPv6 in a network using numerous routing protocols such as RIP, EIGRP, OSPF, IS-IS and BGP, as well as hands-on skills in deploying IPv6 transition mechanisms including various types of tunnels.

You can find additional e-courses in our catalog.

see 1 comments

Common sense prevails over RFC 2328

When trying to extract the OSPF route selection rules from RFC 2328, I've stumbled across a very weird rule (section 16.4.1): if an ASBR within a non-backbone area advertises an external route (or if the forwarding address is within the non-backbone area), it's preferred over external routes advertised by ASBRs in other areas regardless of its metric. I simply had to test this on Cisco IOS … and found out that Cisco engineers prefer common sense to OSPF RFC.

read more see 11 comments

AS-path based filter of customer BGP routes

Any serious (or at least security-aware) ISP should not blindly accept BGP routes from its customers but at the very minimum do some sanity checks on them. For example, if a multi-homed customer is clumsy enough to advertise BGP routes between service providers, it’s nice if you still stop him from turning into a transit AS. The required filter is conceptually quite simple: all the BGP routes from the customer should contain only his AS number in the AS-path.

If you're looking for more in-depth BGP knowledge, try our Configuring BGP on Cisco Routers e-learning solution. If you just need to enhance your hands-on skill, the BGP Remote Lab Bundle is the perfect choice.

read more see 4 comments

Configure OSPF on unnumbered interfaces

When we've been assigning router interfaces in OSPF areas with the network router configuration command, it was impossible to start OSPF only on some unnumbered interfaces and not on others (or place the unnumbered interfaces in different areas). These restrictions are removed if you use the ip ospf area interface configuration command. For example, to put the loopback interface into another area than the WAN links using its IP address, use the following configuration commands:
router ospf 1
interface Loopback0
 ip address
 ip ospf 1 area 0
interface Serial1/0
 ip unnumbered Loopback0
 ip ospf 1 area 1
interface Serial1/1
 ip unnumbered Loopback0
 ip ospf 1 area 2
see 4 comments

IPv6 remote labs on Partner Education Connection

If you need to improve your hands-on IPv6 skills and have access to Cisco's Partner Education Connection, you can get a number of IPv6 remote lab exercises free-of-charge.

Everyone else can buy the same labs from our learning store. For example, for €128, you can get a one week unlimited access to the labs with the ability to repeat every exercise as ofter as you need.

To view the PEC lab content and schedule the labs, just click on one of the following links:

add comment

Gain HIPS skills with e-learning course

Would you like to use the Cisco Security Agent (CSA) v5.0 product to protect host systems from intrusions and security threats but don't have the time to attend classroom training? Do you want to prepare for the CCSP exam? Why won't you learn and practice configuration skills at your own pace in full control of your time?

The Securing Hosts Using Cisco Security Agent (HIPS) e-course is a blended solution that consists of the HIPS web-based training and associated remote labs bundle. It takes a task-oriented approach to teaching the skills needed to deploy, configure, and administer CSA to protect server and workstation hosts.

You can find additional e-courses in our catalog.

add comment

The wonders of machine translations

Today's check of who quotes (and who copies :) my content brought me a "hidden gem": I've discovered a web site that performs automatic translation between any pair of almost 30 languages.

So, if you have nothing better to do and have already checked your daily Dilbert, here's something that will literally make you ROFL: use the machine translation of my blog posts (or any other web page that is technical enough to contain words with "unexpected" meanings) into various European languages. You could start from the form that appears on top of each translated page (just enter URL and select the target language) or you could try the BGP communities post translated into German for starters.

But even if you speak no other language than English, don't despair ... this software does wonders in English-to-English translation as well. Check, for example, my When OSPF Becomes a Distance Vector Protocol post translated to English.
see 3 comments

Interesting QoS problem on Catalyst 3750

Mohammad Faraz Rehan has encountered an interesting problem when using IP access-list based class/policy maps on Catalyst 3750:

When I try to apply the same service-policy to more than 15 interfaces, the configuration command fails and the switch generates the following syslog message:

%QOSMGR-4-POLICER_PLATFORM_NOT_SUPPORTED: Policer configuration has exceeded hardware limitation for policymap …

I've tried to help him with various TCAM-related information and in the end he found an interesting solution to the problem:

It looks like there is a limit related to using the same access-list/class-map/policy-map on multiple interfaces.

The first time I was applying the same policy-map (19 classes/19 ACLs/46 ACL lines) on all interfaces, but the switch would not accept it on more than 15 interfaces. Another test scenario had 18 classes/18 ACLs/52 ACL entries and the same policy-map would only work on 13 interfaces.

Now we implemented 24 different policy-maps, class-maps and ACLs remained the same, and the switch is happy.

see 1 comments

BGP Essentials: BGP Communities

BGP communities are extra attributes you can attach to an IP route carried by BGP. You can use communities to indicate which routes should be propagated or filtered (for example, the well-known NO_EXPORT community signifies that the route it’s attached to shall not be sent outside of the local AS), to influence route selection on remote routers or to trigger other BGP-dependent IOS features (for example, quality-of-service marking based on BGP).

Each BGP community is a 32-bit value. The best practice dictates that the top 16 bits should be the AS number of the network defining the community meaning and the bottom 16 bits are defined by the network administrator.

For example, if you use BGP communities to control QoS marking within your network, the top 16 bits should be your AS number. If you’re using a BGP community to mark backup BGP routes before they are sent to your ISP, you should use the BGP community defined by your ISP (and thus the top 16 bits are the ISP’s AS number).

The only mechanism to set BGP community in Cisco IOS is the set community command in a route-map.

Important: The set community command erases existing communities attached to a route and replaces them with the new set of communities unless you specify the additive option.

You can set BGP communities in any point where you can use a route-map within BGP:

  • on routes you’re receiving from a neighbor with the neighbor route-map in router configuration command;
  • on routes you’re sending to a neighbor with the neighbor route-map out router configuration command;
  • on routes originated into BGP with the network route-map router configuration command;
  • on routes redistributed into BGP with the redistribute route-map router configuration command.

The BGP communities are transitive BGP attribute, meaning that they should be propagated to all BGP neighbors.

Important: Cisco IOS does not propagate BGP communities unless you manually configure community propagation for each neighbor with the neighbor send-community configuration command. BGP peer groups or peer templates are an excellent way to configure BGP community propagation for a large number of peers.

If you're looking for more in-depth BGP knowledge, try our Configuring BGP on Cisco Routers e-learning solution. If you just need to enhance your hands-on skill, the BGP Remote Lab Bundle is the perfect choice.

see 14 comments

When OSPF Becomes a Distance Vector Protocol

A while ago I've read an interesting post where Jeff Doyle, while describing his new-hire interview process, mentioned that inter-area OSPF is actually a distance vector routing protocol. At the time I found the argument interesting, but a bit academic.

A few months later, I've experienced interesting routing flaps in one of my lab setups. A detailed investigation revealed that they were caused by Area Border Routers redistributing inter-area information back into the area from which it originated, effectively creating a temporary routing black hole. You can find the details, in-depth explanation as well as workaround solutions in my february IP Corner article When OSPF Becomes a Distance Vector Protocol.
see 3 comments