Your browser failed to load CSS style sheets. Your browser or web proxy might not support elliptic-curve TLS

Building network automation solutions

6 week online course

reserve a seat

CRS-1 and IOS XR training

NIL’s Professional Services team has helped several customers in IOS XR planning, design and deployment. In almost all cases, the customer’s network engineers were well versed in core Service Provider technologies and Cisco IOS configuration, but lacked specific IOS XR configuration and maintenance knowledge.

Based on the knowledge gap experienced in these projects, our CCIEs have developed CRS-1 and IOS XR training that focuses on the needs of a typical SP network engineer. The course covers the major core Service Provider technologies, including BGP, OSPF, IS-IS, the MPLS suite (MPLS, MPLS TE, MPLS VPN), QoS, IP Multicast and IPv6, always assuming that the students are familiar with the concepts but need IOS XR-specific configuration, monitoring and troubleshooting skills. Extensive lab exercises are spread throughout the course to help students gain hands-on experience with CRS-1 and IOS XR before deploying them in a live production network.

Add comment

Quick tip: limit SNMP traps sent to a SNMP server

If you use multiple SNMP servers in your network, you might want to limit the traps each server receives. Configuring this functionality is easy: just list the traps you want a server to receive at the end of the snmp-server host configuration command.

When you specify the list of SNMP trap types a server should receive, a trap is sent to the server only if it’s listed in the snmp-server host command and enabled with the snmp-server enable traps command.

see 2 comments

OSPF router ID selection: all the details

After I wrote the OSPF router ID selection trivia post I wanted to figure out all the details of the OSPF router ID selection algorithm. As I’ve expected, the common wisdoms are mostly correct, but they fail to cover the interesting border cases. The complete algorithm (as seen from the outside) is documented in the “OSPF Router ID selection algorithm” article.

Read the whole article in the CT3 wiki

Add comment

SPF events in OSPF and IS-IS

The Shortest Path First (SPF) algorithm used by all (read: both) popular link-state routing protocols should be well-known to all network engineers, but the fact that the IP routing table gets populated by a distance-vector-like second phase of the algorithm is oft forgotten.

Check this short presentation in case you’d like to get a bit more in-depth explanation of the SPF algorithm.

The second phase of the link state route selection algorithm is called Partial SPF in OSPF and Partial Route Calculation (PRC) in IS-IS. Obviously it’s beneficial if the router can react to a change in the network topology by running PRC, not the full SPF, and there are significant differences in the way OSPF and IS-IS respond to various topology change events. The differences are not architectural; you could make OSPF behave as well as IS-IS. They just prove that the old wisdom is still true: very large IOS-based networks use IS-IS because it’s better implemented than OSPF.

Read the SPF events in OSPF and IS-IS article in the CT3 wiki

Add comment

OSPF Router ID selection trivia

True or false: If you have an (enabled) loopback interface configured on the router, its IP address will always be used as the OSPF router ID.

Answer: False. Each OSPF process running on the router needs a unique Router ID. If you have a single loopback interface and two OSPF processes, the second OSPF process is forced to select another interface address as its router ID.

Here is a sample router printout:

Test#show running | inc ^interface|^ ip address
interface Loopback0
 ip address
interface FastEthernet0/0
 ip address
Test#show ip ospf | inc Routing Process
 Routing Process "ospf 2" with ID
 Routing Process "ospf 1" with ID
see 8 comments

Service Control Engine Operations Training

The CCIE gurus in NIL’s training department have recently developed a brand new Service Control Engine (SCE) Operations hands-on training. The course covers the real-life needs of network engineers deploying SCE-based solutions. It provides numerous design solutions and case studies and does not focus exclusively on SCE but also on the associated Subscriber Manager (SM), Quota Manager (QM) and Collection Manager (CM) servers. The hands-on labs allow you to practice standard operational tasks, including traffic policy creation, subscriber operations and maintenance tasks.

Add comment

Quick tip: Enable SNMP traps

IOS can generate numerous SNMP traps, but you have to enable most of them manually. Configuring a server that receives SNMP traps with the snmp-server host address community traps is not enough; you have to enable individual trap categories with the snmp-server enable traps group [ trap ] command.

In older releases, the standard SNMP traps (for example, link up/down traps) are enabled by default and cannot be disabled with the snmp-server enable traps command. Newer IOS releases have added the snmp-server enable traps snmp [ authentication | coldstart | linkdown | linkup | warmstart ] global configuration command. These releases require you to enable standard SNMP traps manually; otherwise the router will not generate them.

If you want to see the whole list, configure snmp-server enable traps and execute show running | include snmp-server enable. IOS 12.4T network management reference guide also includes a comprehensive list of all options.

see 2 comments

Symmetric VPN routing triggered from the central site

A few days ago, Brent asked a really interesting question in NIL’s forum:

I am wondering if there is a way to change routing preference from using one VPN link to another VPN link, and making this change only at one side (head end) of the bundled vpn's and have the traffic path still remain symmetric.

One of the solutions I’ve proposed (using BGP with weights and MED on the central router) worked really well for him.

If you have an interesting network-related problem, you might consider using NIL’s forums to ask NIL’s CCIEs for an opinion. If you’re new to BGP and would like to use it in your enterprise network, start with the BGP resource center in the CT3 wiki.

Add comment

Carrier Ethernet: the big picture

My regular readers are probably not surprised to learn that I dislike buzzwords and marketectures. I’ve heard so much about Carrier Ethernet in the last few years (not including confusing definitions of what it’s supposed to be) that I decided to write an article about it. It was recently published by SearchTelecom as “Carrier Ethernet: Big picture issues for carrier deployment”.

Add comment

Why are there no Untagged entries in my LFIB?

One of the readers of my “When would an MPLS LSR have Untagged output label?” post made an interesting comment:

When a loopback network is advertised as, it's seen as »pop tag« on the neighboring router and I can't see it in the »show mpls forwarding« printout on the local router. What's going on?

As explained in the “When would an MPLS LSR have Untagged output label?” post, the Untagged (also displayed as No label in recent IOS releases) value means that the Label Switch Router (LSR) cannot use the inbound label to decide what to do with the packet and has to perform layer-3 lookup.

In most cases, an LSR using MPLS for IP routing can advertise an implicit null label to its upstream neighbors to indicate it wants to receive an IP packet, not a labeled packet. The upstream LSR performs Penultimate Hop Popping (PHP) as indicated by a Pop tag (recently Pop Label) entry in its LFIB. The LSR we’re discussing has just saved a lookup operation, as it doesn’t have to perform label lookup followed by an IP lookup.

In these cases, the LSR is allocating labels to IP prefixes only if it has received corresponding labels from downstream LSRs. The untagged IP prefixes thus never appear in the Label Forwarding Information Base (LFIB) and in the show mpls forwarding display.

Sometimes a router must receive a labeled packet from the upstream LSR (PHP is not an option). There are two well-known scenarios: Label-controlled ATM (LC-ATM) where the MPLS label is part of ATM cell headers and MPLS VPN, where the inbound label (also) indicates the VRF to use. In both cases, you’ll see No label entries in LFIB for directly connected or aggregate routes. In the LC-ATM case, you’ll also see No label entries associated with all IP destinations for which the next-hop router has not advertised a label.

This article is part of You've asked for it series.

see 3 comments

True or false: MPLS VPNs offer equivalent security to Frame Relay

Years ago when vendors were pushing the MPLS story to Service Providers, an “independent analyst” wrote a report claiming that MPLS-based VPNs offer security equivalent to Frame Relay networks (to find those reports, ask Google about “MPLS security equivalent to Frame Relay”). This might be true from the functional perspective (and it’s absolutely true that using IP does not make MPLS-based VPNs inherently insecure), but anyone believing these reports might become mightily upset when learning about BGP and MPLS security issues.

Before going any further, please note that exploiting the MPLS architecture as described in All your packets are belong to us presentation requires access to the core SP network, which means that the network (or network management station) has already been successfully penetrated.

A perfectly implemented MPLS VPN network running on equipment with no vulnerabilities is indeed at least as secure as a Frame Relay network … but it’s much easier to implement a secure Frame Relay network (well, it’s very hard not to, unless you’re clumsy enough to configure a customer port as NNI port) than deploying all security measures on PE routers. There are also a few other details that make the life of the MPLS VPN security gurus a bit harder:

  • The only protocol running between a Frame Relay edge switch and a CE router is LMI. LMI is so simple it’s almost impossible to make it vulnerable. Protocols running between PE- and CE-routers are more complex.
  • A Frame Relay switch cannot accept a layer-3 packet from the customer equipment. Any IP host can send packets address to the PE-router.
  • Network management protocols in Frame Relay network are different from what the customers are using. MPLS VPN network uses the same management protocols as the customer’s equipment using the same layer-3 transport.
  • IP requires multiple auxiliary protocols (ARP, ICMP …) to work correctly. Any one of these protocols can be used to cause a denial-of-service attack against a PE-router, sometimes even from an end-host. The only harm you could do to a Frame Relay switch is to send too many LMI packets from the CE-router.

Last but definitely not least, some Service Providers offer MPLS VPN services on the same equipment that offers Internet access through the global IP routing table. The PE-routers in these networks are significantly more exposed than the dedicated PE-routers.

Faced with all this, what can an end-user do? There are a few simple actions you can take:

  • When transporting sensitive data across an MPLS VPN network, don’t rely on a third party (SP) to protect it. Use IPSec.
  • When you’re concerned about reliability and/or uptime, use multiple service providers.
  • Don’t believe that you can get exactly the same level of security when using a service that offers 100-times the bandwidth for one tenth of the price of the alternative.
Add comment

I hope you knew this already: Backbone-hacking tools

A few days ago (the) Steve Bellowin sent an e-mail to the NANOG mailing list pointing to a FUD-full article describing upcoming release of MPLS hacking tools. Christian Koch quickly pointed out a similar presentation given by the same group @ Schmoocon and numerous respondents correctly stated these are old issues … if you’re interested in BGP and MPLS security, of course. Nicolas Fischbach even provided link to a 7 year old presentation describing numerous BGP/IGP/MPLS risks and attack vectors.

I went through the presentation and it does not describe anything we wouldn’t have known already. The capability to fake platform-wide MPLS labels has been known since the times MPLS (then known as Tag switching) was running mostly in PowerPoint. We’ve thus always recommended that the Service Providers should not run MPLS with their customers.

Carrier’s carrier architecture is a different story; it separates VRF labels from global labels.

The Carrier Ethernet attacks described in the presentation are also pretty obvious. If someone is brave enough to have a bridged solution spanning the globe (lots of people got burnt doing that 15 years ago, but obviously it’s best to learn from your own mistakes), he’s literally asking an intruder breaking into one of the sites to attack other sites with L2 techniques.

By now you might wonder why I’m writing this post. While the “All your packets are belong to us” presentation does not describe new vulnerabilities, it’s still a nice eye-opener for someone who thought SP MPLS infrastructure is absolutely secure (more about that in tomorrow’s post). It’s not and if you’re an enterprise network with high security requirements, you should take your own precautions, including running IPSec over SP MPLS network.

see 4 comments

IOS interface names

George sent me a question that surfaced age-old memories:

I saw the Serial 0/1/0 interface in one of your articles. I understand the Serial 0/1 command as accessing the sub interface of Serial 0 with the 1st interface. But I have never seen the 2nd 0 being used. What is the 2nd "0", and how is it to be used?

In the ancient times when the high-end router was an AGS+, the interface names were kept simple (for example, Serial0). When the Cisco 7000 was introduced with online insertion and removal (OIR) capability, router's life became more complex, as its actual hardware (and thus the interface names) might change while it's running.

To work around this problem, the interface names changed with Cisco 7000 (and broke quite a few of our scripts). Every interface name on that router had two parts: the slot number and the port-in-slot number. Later the same convention was adopted for most modular routers (it doesn’t make sense to have all your interfaces renumbered if you exchange a 2-port WAN card with a 4-port WAN card) and with the addition of WIC (Wan Interface Card) adapters which are effectively module-on-a-module, we’ve got the third layer of interface hierarchy. The Serial2/1/0 interface is thus port#0 on WIC#1 on NM#2.

There are (at least) two other special characters in the interface names:

  • Dot (.) indicates a subinterface. Serial0/1/0.101 is subinterface 101 on WAN interface Serial0/1/0.
  • Colon (:) indicates a channel or channel group on a channelized interface (BRI, E1, T1 …).

If you’re aware of any other interface name format, please take your time to write a comment.

This article is part of You've asked for it series.

see 7 comments

Book review: IPv6 Security

The high-end books published by Cisco Press are usually pretty good, but every now and then they manage to produce a masterpiece that has all the potential to become a legend. The “IPv6 Security” book by Scott Hogg and Eric Vyncke is definitely in this category and is a must-read for anyone who plans to deploy IPv6 in the future (that should include around 100% of the network engineers).

read more Add comment

Secondary subnets limitation

Chris sent me an interesting question:

How many secondary IPs can you put on a Vlan on a Catalyst switch?

The best way to figure out the answer to this question is to close the browser window pointing to (you won’t find the answer there), generate a test configuration and try to load it into your box.

I’ve written a one-line PERL script that generated the ip address secondary commands for me (you could also do it in almost any tool, including Excel) …

perl -e 'for ($i=1;$i<255;$i++){
  print "ip address 10.22.$i.1 secondary\n" }'

… and pasted the results into the console window. No problem, a router accepted at least 250 secondary addresses. Chris repeated the process on his Catalyst switch and reported that he stopped the test after approximately 150 addresses (obviously way more than what he needed).

Remember: experiment is sometimes the shortest path to the solution (a fact already known to ancient Greeks).

This article is part of You've asked for it series.

see 3 comments

Create numerous BGP sources with a single router

Sometimes you need numerous BGP sources in your lab, but have only a fixed number of routers. Although you could introduce an extra BGP source with Quagga running on Linux (or use tricks to generate more than one BGP source on a single Linux host), it’s usually best if you could avoid the introduction of extra devices in your lab.

Recent releases of Cisco IOS provide a perfect solution: you can run as many BGP instances as you wish on a single router (each one in its own VRF) and use the BGP Support for Dual-AS Configuration to replace actual AS number with the desired one.

Read the whole article in the CT3 wiki

see 4 comments

BGP Route Reflector update groups (technical details)

One of the readers of my BGP Route Reflectors article spotted an obvious deviation from “how we always though the route reflectors work”. A lab test confirmed the validity of my claims: a BGP route reflector does send an update back to the client from which it was received (and it’s perfectly legal according to the new BGP Route Reflector RFC).

Read the whole article in the CT3 wiki

This article is part of You've asked for it series.

Add comment

Another green initiative from Cisco Systems

Cisco has unveiled numerous green solutions in the last few months, including energy-saving ASR 9000 router and GreenWise initiative. Now they went a step further and implemented some of the old energy-efficient wireless technologies, proving that you don’t always need new technologies to address the energy efficiency (see also Myth #8 in Top 10 Myths about Sustainability).

The first step in GreenWise wireless rollout are the new high-density line cards for the ASR 9000 routers. The port adapters and SFP modules in these line cards will support RFC 1149 with its low-electricity requirements, low-emission focused RFC 1926 as well as RFC 4824 addressing the needs of the health care vertical market. The first release of the line cards will not support all QoS requirements of RFC 2549 due to the limited processing power of the on-card PPU.

You can find detailed information on these revolutionary line cards and adapters among the other in-depth data sheets documenting ASR 9000 platform and line cards (snapshot of the page).

see 4 comments