Category: MPLS VPN

End-to-end troubleshooting of MPLS VPN solutions is one of the more complex network troubleshooting tasks. On top of several sophisticated technologies and protocols used in MPLS VPN solutions, we have to deal with customer-to-provider interaction on the IP routing protocol level, which makes the troubleshooting efforts even more convoluted.

To minimize the impact of your customers on your troubleshooting efforts, you might want to start with the PE-to-PE troubleshooting. When used as the first step in your troubleshooting process, the PE-PE tests will bypass customer errors, intra-site customer routing problems, PE-CE interactions, and route redistribution issues.

It’s obvious why two routers in the same OSPF domain cannot have the same router ID. However, requiring unique router IDs on OSPF processes running in different VRFs is probably too harsh, even though it does prevent confusion if two VRFs ever get connected through a customer site. Anyhow, if you have overlapping IP addresses on loopback interfaces in different VRFs, OSPF process might not start.

Deploying EIGRP as the PE-CE routing protocol in MPLS VPN networks is easy if all sites have a single PE-CE link and there are no backdoor links between the sites. Real life is never as simple as that; you have to cope with various (sometimes undocumented) network topologies. Even that would be manageable if the customer networks would have a clean addressing scheme that would allow good summarization (that happens once in a blue moon) or if the MPLS VPN core could announce the default route into the EIGRP sites (wishful thinking; the customer probably has one or more Internet exit points).

We described EIGRP-in-VRF in MPLS and VPN Architectures, Volume II. A few details have changed in the meantime; you have to configure the following features to get EIGRP running within MPLS/VPN environment:

• The autonomous-system command within the VRF address family is mandatory, even if the VRF AS number matches the EIGRP process number.
• The default BGP-to-EIGRP redistribution metric has to be configured, otherwise remote EIGRP routes will not be redistributed even though they have EIGRP metric encoded in extended BGP communities.
• Things work best if you disable auto-summary on PE-routers.

If you use multiprotocol BGP (MP-BGP) in your network to support MPLS VPN, IPv6 or IP Multicast over BGP, it's best if you go all the way and configure an explicit ipv4 address family; the resulting BGP configuration is significantly easier to read and understand as the session-specific parameters are clearly separated from the routing-specific parameters and the IPv4 settings are nicely grouped in an explicit section.

To change the format of the BGP configuration, configure the IPv4 address family with the address-family ipv4 unicast router configuration command (the neighbor statements and other configuration settings pertinent to IPv4 configuration are automatically moved into the new address family) or manually activate a BGP neighbor for IPv4 route exchange with the neighbor activate router configuration command.

Cheng sent me an interesting question:

I'm reading your book MPLS and VPN Architectures and I've found the ip vrf forwarding name fallback global command in the “Additional Lookup in the Global Routing Table” section. I can only find this command in Junos, but not in IOS.

… and he was right. When we were writing the book, we described several features that were still in development as it looked like they would be in the production code by the time the book was published. Many of them made it into the public IOS releases (for example, the Carrier's Carrier architecture), but some of them (like this command) simply vanished from the surface.

However, it looks like the engineers that switched from Cisco to Juniper took the concept with them and implemented it in JunOS, so JunOS has this feature but IOS doesn't.

This article is part of You've asked for it series.

Occasionally someone would try to persuade me that the layer-2 VPN services are like aspirin (you know, totally harmless plus it could get rid of all your headaches). OK, that might be true if you take the layer-2 VPN offering as a pure transport solution and plug in an extra router (sometimes also called a layer-3 switch by marketing people) between the Service Provider’s Ethernet (or whatever they give you) and your LAN. But there are people who don’t know the details and plug the SP Ethernet straight into their L2 switch … and things might even work for a while … until the whole network collapses.

In my opinion, we need both L2 and L3 VPN services, but it’s important that they are positioned and deployed correctly. You can read more about my views on this topic in the SearchTelecom article Making the case for Layer 2 and Layer 3 VPNs.

An IOS device configured for IP+MPLS routing uses three different Maximum Transmission Unit (MTU) values:

• The hardware MTU configured with the mtu interface configuration command
• The IP MTU configured with the ip mtu interface configuration command
• The MPLS MTU configured with the mpls mtu interface configuration command

The hardware MTU specifies the maximum packet length the interface can support … or at least that's the theory behind it. In reality, longer packets can be sent (assuming the hardware interface chipset doesn't complain); therefore you can configure MPLS MTU to be larger than the interface MTU and still have a working network. Oversized packets might not be received correctly if the interface uses fixed-length buffers; platforms with scatter/gather architecture (also called particle buffers) usually survive incoming oversized packets.

IP MTU is used to determine whether a non-labeled IP packet forwarded through an interface has to be fragmented (the IP MTU has no impact on labeled IP packets). It has to be lower or equal to hardware MTU (and this limitation is enforced). If it equals the HW MTU, its value does not appear in the running configuration and it tracks the changes in HW MTU. For example, if you configure ip mtu 1300 on a Serial interface, it will appear in the running configuration as long as the hardware MTU is not equal to 1300 (and will not change as the HW MTU changes). However, as soon as the mtu 1300 is configured, the ip mtu 1300 command disappears from the configuration and the IP MTU yet again tracks the HW MTU.

The MPLS MTU determines the maximum size of a labeled IP packet (MPLS shim header + IP payload size). If the overall length of the labeled packet (including the shim header) is greater than the MPLS MTU, the packet is fragmented. The MPLS MTU can be greater than the HW MTU assuming the hardware architecture and interface chipset support that (and the router will warn you that you might be getting into trouble). Similar to the ip mtu command, the mpls mtu command will only appear in the running configuration if the MPLS MTU is different from the HW MTU. However, contrary to the behavior of the IP MTU, any change in HW MTU with the mtu configuration command also resets the MPLS MTU to HW MTU.

The behavior as described above was tested on a 3725 router running IOS release 12.4(15)T1. Although the MPLS MTU Command Changes document claims that you cannot set MPLS MTU larger than then interface MTU from IOS release 12.4(11)T, I was still able to do it in 12.4(15)T1.

I wanted to get in-depth details on how various MTU parameters interact in GRE/IPSec/MPLS environment. Before going into router configuration details, I wanted to have a tool that would reliably measure actual path MTU between the endpoints. After a while, Google gave me a usable link: supposedly the tracepath program on Linux does what I needed. As I'm a purely Windows user (for me, PCs are just a tool), I needed a Windows equivalent … and found mturoute, the utility that does exactly what I was looking for.

The MPLS VPN implementation on Cisco IOS has always allowed you to create VRF static routes that pointed to interfaces belonging to other VRFs. The feature can be used to implement interesting overlapping VPN (or common services VPN) designs, some of which are explained in the MPLS and VPN Architectures books.

However, quite often the ability to create inter-VRF static routes is considered a major security problem, as an operator configuration error could establish undesired inter-VPN connectivity. In these cases, use the no ip route static inter-vrf configuration command to prevent such routes from being installed in the VRF routing table.

You might also want to read a good explanation of MPLS VPN route leaking from Cisco systems

When we were writing the MPLS and VPN architectures books, there was a limit on the number of OSPF processes you could configure per PE-router. The limit was based on the fact that IOS supports up to 32 routing information sources. Two of them are static and connected; you also need an IGP and BGP in the MPLS VPN backbone, resulting in 28 OSPF processes that could be configured on a single PE router. This “feature” severely limited OSPF-based MPLS VPN deployments until IOS release 12.3(4)T when the limitation was removed, resulting in the availability of up to 30 routing processes per VRF.

RIP, BGP, and EIGRP never experienced the same limitations as you configure VRF-specific routing instances within address families of a single routing protocol

Following my post about the relationship between the MPLS and VPN architectures books and CCIP MPLS exam, Peter Dob had an excellent idea: combine the MPLS and VPN architectures (Volume I, CCIP edition would be even better) with the MPLS fundamentals from Luc de Ghein. By reading Luc's book, you'll also get exposure to other MPLS-related topics (for example, AToM) on top of MPLS TE overview that you need for the exam.

This article is part of You've asked for it series.

Every now and then I'm getting questions from my readers regarding the suitability of my MPLS books for the CCIP exam, for example:

I'm pursuing my CCIP and have a hard time finding the right MPLS study guide. I know you have the CCIP edition that was written in 2002, but I think the exam topics have changed. Can you recommend what book or books are best for the CCIP MPLS exam?

Are MPLS VPN Architectures Volume 1 & 2 two completely separate books or is Volume 2 a newer release. I was thinking of going for the CCIP and wanted to know if I should get both books or just the more recent one.

IOS release 12.3(11)T introduced Half-duplex VRF, a great feature for those of us who have to implement hub-and-spoke VPN (the VPN where all traffic has to pass through the central site), but hate the configuration hassle associated with it. Unfortunately, the way this feature is implemented, you can only configure it on virtual access/template interface, making it useless in most access networks. Too bad ...