Russell Heilling made a highly interesting observation in a comment to my MPLS MTU challenges post: you could get asymmetric MTUs in MPLS networks due to penultimate hop popping.
The only link using MPLS is between R1 and R2. FW is a misconfigured firewall blocking all ICMP packets. Furthermore, FW uses NAT making the client C appear to be directly connected to R2. Layer-2 payload size (known as MTU on Cisco IOS) on all links is 1500 bytes. Unlabeled IP packets can be up to 1500 byte long; labeled IP packets cannot exceed 1496 bytes (depending on the size of the MPLS label stack).
R1 and R2 advertise labels for all known prefixes to each other using LDP. R1 advertises a “real” label for S (because it’s reachable through a next-hop router); R2 advertises implicit null label for FW/C to R1 to enable penultimate hop popping.
When the server S sends a packet to client C, R1 should send a labeled packet to R2, but due to implicit null advertised by R2, the MPLS label stack is empty; IP MTU from S to C is 1500 bytes.
When C sends a packet to S, R2 inserts a single MPLS label in front of the IP payload (remember: R1 advertised a non-null label for S to R2); IP MTU from C to S is 1496 bytes.
In properly configured networks, asymmetric MTUs wouldn’t matter; combined with misconfigured firewalls they can be fatal and extremely hard to troubleshoot. In our scenario, client would be able to download any content from the server (unless the HTTP request header or its equivalent grows beyond 1456 bytes), but would fail to upload anything longer than ~1400 bytes.
To learn more about MTU path discovery and related problems, read the Never-Ending Story of IP Fragmentation.
You’ll find in-depth description of MPLS/VPN technology and enterprise network deployment hints in our Enterprise MPLS/VPN Deployment. For more VPN webinars, check our VPN webinar roadmap. You get access to all those webinars when you buy the yearly subscription.