End-to-end troubleshooting of MPLS VPN solutions is one of the more complex network troubleshooting tasks. On top of several sophisticated technologies and protocols used in MPLS VPN solutions, we have to deal with customer-to-provider interaction on the IP routing protocol level, which makes the troubleshooting efforts even more convoluted.

To minimize the impact of your customers on your troubleshooting efforts, you might want to start with the PE-to-PE troubleshooting. When used as the first step in your troubleshooting process, the PE-PE tests will bypass customer errors, intra-site customer routing problems, PE-CE interactions, and route redistribution issues.

In one of the MPLS-related posts, I’ve described the role of implicit NULL in penultimate hop popping (PHP). To make the distinction between implicit and explicit NULL even clearer, I’ve prepared a short explanation with corresponding diagrams.

It looks like everyone has better things to do after the summer holidays than providing free content to the blogosphere. The only really interesting post I found was Colin McNamara's response to my DMZ VLAN post, in which he looks at the much bigger picture: it's not only the switching infrastructure we're sharing; in the virtualization rush, we're sharing memory, disks, interfaces and even firewalls.

read more add comment

The RFC 3101 (OSPF NSSA Option) states:

In addition, an NSSA border router should originate a default LSA (IP network is 0.0.0.0/0) into the NSSA. Default routes are necessary because NSSAs do not receive full routing information and must have a default route in order to route to AS-external destinations.

I am pretty sure IOS inserted the type-7 default route into an NSSA area when the NSSA feature was introduced.

The Thinking problem management! blog had an interesting article on The Leaky VLANs myth, quoting a test report from SANS Institute that documents how you can inject frames into other VLANs even if you're not connected to a trunk port. The report is eight years old (so one would hope this issue has been fixed in the meantime), but there's another question you should ask yourself is: what happens when you lose the configuration of the switch (and I've seen devices losing configuration after a power glitch). If you're using a router to perform L3 switching, no harm is done; a router with empty configuration forwards no packets. But if you're using a low-end switch, you're in deep trouble; by default, a switch forwards packets between all ports ... and if you use static IP addresses on all subnets, you won't even notice they're connected. If you want to be very safe, you're better off having a different set of switches for the inside and the outside zones of your firewall.

If you’ve ever had the “privilege” of buying equipment from a large systems integrator (or directly from a large vendor), you’re probably familiar with this process:

A while ago I've documented how you can cope with DHCP clients that do not send Client identifier (DHCP option 61) in their DHCP Discover/Request messages, but some people are still trying to persuade me that the client-identifier pool configuration command should work. I really wanted to be sure I hadn't missed something, so I started Wireshark and captured the actual DHCP Discover packet generated by a Linux host:

A week ago I've asked whether anyone has a good use for BGP aggregation. Out of 1300 daily visitors of my blog and 2200 subscribers to the RSS feed,  four (4) people found the time and energy to reply. Thank you for your thoughts!

Let me conclude that BGP aggregation is not widely used (so I will spend much energy covering it in my future posts) ... all the other potential conclusions are too pessimistic.

One of the presentations at the recent Defcon 16 event demonstrated how you can use the very common laziness of the Internet Service Providers to hijack any prefix you want (just ask YouTube). Nothing new so far, but the part where they fake the AS path in the hijacked announcement to create a safe (hijack-free) conduit back to the destination is brilliant ... and the TTL manipulation is the icing on the cake.

Of course this is a well-known BGP vulnerability (actually, implementation sloppiness on the part of ISPs) that we've been writing about for a long time, but the Defcon presentation is probably the first documented step-by-step recipe for a realistic MITM attack.

Hat tip to Jeremy Stretch; I found the link to the Defcon presentation on his blog.

read more see 16 comments

In the Deploying Zone-Based Firewalls book I wrote:

In early releases supporting zone-based policy firewall configuration (IOS 12.4(6)T), match protocol command cannot be used to classify traffic to or from the self zone. Only IP access lists can be used for traffic classification purposes.

Misha Volodko reported that the match protocol icmp command works for him when used with the self zone. Another small step toward perfect implementation :) ... and don't forget that you can always use class class-default to catch all the unclassified traffic (and log it before it's dropped, for example).

Establishing parallel EBGP sessions across parallel links between two edge routers (EBGP peers) – as displayed in the diagram below – is the most versatile form of EBGP load balancing. It does not require static routing or extra routing protocol (like the design running EBGP between routers’ loopback interfaces), device-specific tricks like configuring the same IP address on multiple interfaces) or specific layer-2 encapsulation (like Ethernet LAG or Multilink PPP).

It even allows proportional load-balancing across unequal-bandwidth links and combinations of various layer-2 technologies (for example, load-balancing between a serial line and an Ethernet interface). The only drawback of this design is the increased size of the BGP table, as every BGP prefix is received from the EBGP neighbor twice.

Ralf sent me a SNMPv3 question:

If I create a SNMPv3 user which has a password (snmp-server user userthree groupthree v3 auth md5 user3passwd), this user does not appear in the running- or startup-config. Cisco even documents this if you know what to look for.

I strongly suspect (although I did not test this) that these users are also missing from configuration exported to TFTP servers. What would be the recommended way to make usable config backups of routers with such users?

Like certificates, the SNMPv3 users are stored in private-config and thus never appear in the router configuration. If you want to have a backup of the user data, create a text file on one of your NMS servers, add SNMPv3 usernames and passwords in the text file and use the copy somewhere running-config to configure SNMPv3 users on the routers.

Peter Weymann sent me a really intriguing question:

A few days ago I started reading the Ciscopress book End-to-End Network Security: Defense-in-Depth and stumbled over the scheduler command. This one could be used to allocate time that the cpu spends on fast switching packets or process switching packets, if I understand it correctly. They also mention interrupting CPU processes but honestly I don't really understand how it works.

Cisco routers support (at least) three forms of layer-3 switching (formerly known as routing). CEF switching and fast switching are performed entirely within the interrupt context (I/O adapter interrupts a process the CPU is currently executing and all the work is done before the process resumes). Process switching is performed in two steps: packet is briefly analysed within the interrupt context and requeued into the IP Input process where it's eventually switched. Almost all I/O adapters used these days use a concept of RX/TX rings to communicate with the CPU, meaning that the CPU potentially has to handle more than one packet for each interrupt.

Fast switching is gone starting with IOS release 12.4(20)T.

Under very high load, the packet arrival rate could be so high that the router would constantly service packets within the interrupt context without ever returning back to the IOS processes.

You can check the CPU load incurred by the interrupt context and IOS processes with the show process cpu command. The second number in the five seconds part of the first line tells you the amount of interrupt context activity in the last five seconds.

To prevent the starvation of IOS processes (which could result in keepalive and routing protocol problems, eventually leading to loss of routing protocol neighbors), the scheduler allocate command limits the amount of time that can be spent in the interrupt context and allocates some guaranteed time to the IOS processes. Very probably the routers have a mechanism to mask the requests from the I/O adapters during that period so that the CPU is not interrupted (BTW, this slightly increases the jitter).

A similar command is the scheduler interval command. IOS has high- and low priority processes. Whenever the CPU has to decide what process to run (usually following an interrupt or when a process decides it's done with its work), it will run a high-priority process if one is ready. This could lead to starvation of low-priority processes and the scheduler interval command specifies the maximum amount of time the higher-priority processes can consume before a low-priority process is given a chance to run.

Unless you have serious (and I mean __serious__) problems in your network, don't play with these commands. They are a last-resort things you can do if you're under very heavy load and still need access to the exec to reconfigure the router. In most cases, you should not have to worry ... and anyhow, if the CPU load is close to 100%, you have other problems anyway.