The MPLS MTU challenges

@MCL_Nicolas sent me the following tweet: “Finished @packetpushers Podcast show 7 with @ioshints ... I Want to learn more about Mpls+Mtu problem” You probably know I simply have to mention that a great MPLS/VPN book and a fantastic webinar describe numerous MPLS/VPN-related challenges and solutions (including MTU issues), but if MTU-related problems are the only thing standing between you and an awesome MPLS/VPN network, here are the details.

The moment you configure mpls ip on an interface, several things happen in the background (assuming MPLS hasn’t been configured on the router before and you haven’t changed default settings):

  • LDP process is started and starts sending LDP hellos (UDP packets on port 646) on all MPLS-enabled interfaces;
  • Labels are assigned to all non-BGP IP prefixes in the CEF table and stored as incoming labels in Label Forwarding Information Base (LFIB). The outgoing action for all those labels is untagged (remove the top label, check the label stack is empty, and forward the IP packet).

When a router finds an LDP neighbor, it:

  • Establishes an LDP session (TCP session between loopback interfaces);
  • Exchanges all assigned labels with the LDP neighbor and stores labels sent by the neighbor in the Label Information Base (LIB);
  • Inserts outgoing label for every IP destination in LFIB where the LDP neighbor is the IP next hop.
  • Inserts MPLS label stack (with a single label advertised by the LDP neighbor) in the CEF table for every IP destination where the LDP neighbor is the IP next hop.

Summary: using the default settings in Cisco IOS, whenever two routers establish an LDP session, all IP packets forwarded between them become labeled packets with an MPLS label stack.

The default IP MTU on all router interfaces is 1500 bytes. The default MPLS MTU is the same, but as a single-label MPLS label stack takes 4 bytes, the MTU for labeled IP packets has to be reduced. The two routers can still exchange 1500-byte IP packets as long as they are not labeled. Labeled IP packets cannot be longer than 1496 bytes (remember: all forwarded packets are labeled if you use default settings).

Enabling MPLS between two routers does not affect the end hosts – their MTU is still 1500 bytes and their packets will be fragmented by the ingress Label Switch Router (LSR) ... unless the hosts use MTU path discovery, in which case the now oversized packets will be dropped and ICMP messages will be sent back to the end hosts.

MTU path discovery usually works well in enterprise network, unless an overzealous security administrator decides to configure a firewall (or a router ACL) to drop all ICMP packets. A networking engineer misapplying best-practices documents and configuring no ip unreachables has the same effect. In both cases, path MTU discovery is broken and most of the application traffic crashes when you enable MPLS (VoIP might not be affected due to small packet sizes it uses).

More information

To learn more about MTU path discovery and related problems, read the Path MTU Discovery article written by Jeremy Stretch and my Never-Ending Story of IP Fragmentation.

You’ll find in-depth description of MPLS/VPN technology and enterprise network deployment hints in my Enterprise MPLS/VPN Deployment webinar (register for a live session or buy its recording). For more VPN webinars, check my VPN webinar roadmap. You get access to all those webinars when you buy the yearly subscription.

7 comments:

  1. Russell Heilling15 July, 2011 08:12

    "Summary: using the default settings in Cisco IOS, whenever two routers establish an LDP session, all IP packets forwarded between them become labeled packets with an MPLS label stack."

    This is a bit of a simplification (understandable in a short summary article ;) ). There is another default behaviour that can cause confusion around MTUs...

    By default local connections will be advertised with the "implicit null" label, which will result in penultimate-hop-pop (PHP) behaviour. When you have PHP taking place and an MTU constraint you can end up with paths that have asymmetric MTUs.

    This can be real fun to troubleshoot if you aren't expecting it. :)

    ReplyDelete
  2. Hi Ivan, i recently happen to encounter an issue with cisco ios, if we enable the command "mpls bgp forwarding" under an interface, "ip tcp adjust-mss" command has no effect!

    are they any solutions to this problem and what exactly does "mpls bgp forwarding" do?it also had "mpls mtu max" under the interface.

    Thanks!

    ReplyDelete
    Replies
    1. Were you running BGP over that interface? This might explain what that command does (or you could google for it):

      http://www.cisco.com/en/US/docs/ios/12_0s/feature/guide/fsiasleb.html

      Delete
  3. Hi Ivan, excellent blog! I'm enabling mpls in our Cisco network (7609s,7206 and 7204 routers). When I enable mpls between 7609 (it has access to internet, and it's our core router) and 7206 (our distribution/access router), internet traffic is dropped. When I see which internet traffic is affected, It's TCP traffic. I'm pretty sure that UDP traffic is OK because I enabled mpls between 7206 and 7204 and transport voice traffic only(UDP), without problems. It could be a MTU issue? I have mpls mtu and ip mtu in 1500.

    ReplyDelete
    Replies
    1. It's very probably MTU issue (although I would check for any error messages on all boxes involved). VoIP packets are very small, so they're not affected by that.

      The "MPLS MTU" actually specifies the maximum MTU of a labeled packets, so you need to increase that to 1508. See also http://www.ciscopress.com/articles/article.asp?p=680824&seqNum=5

      Delete
    2. Hi Ivan, I checked my router logs (as you told me) and I found this message: "mpls_lookup_inline: dropping mpls packet (0x6C1B8100 0x20AE8847 ns 0x0001713D 0x4500003C) - no subblock for FastEthernet0/0/FastEthernet0/0.777". In cisco documentation I couldn't find the meaning of this message, any clue?

      Delete
    3. No idea. Time to open a case with Cisco TAC.

      Delete

You don't have to log in to post a comment, but please do provide your real name/URL. Anonymous comments might get deleted.

Ivan Pepelnjak, CCIE#1354, is the chief technology advisor for NIL Data Communications. He has been designing and implementing large-scale data communications networks as well as teaching and writing books about advanced technologies since 1990. See his full profile, contact him or follow @ioshints on Twitter.