All MTUs are not the same

Matthew sent me the following remarkable fact (and he just might have saved some of you a few interesting troubleshooting moments):

I was bringing up an OSPF adjacency between a Catalyst 6500 and an ASR 9006 and kept getting an MTU mismatch error. The MTU was set exactly the same on both sides. So I reset them both back to default (1500 on the 6500 and 1514 on the ASR 9006) and the adjacency came back up, even though now the MTU is off by 14 bytes. So I attempted to bump the MTU up again, this time setting the MTU on 6500 to 1540 and the MTU on the ASR 9006 to 1554. Adjacency came right up. Is there something I am missing?

The 14 byte difference is the crucial point – that’s exactly the L2 header size (12 bytes for two 6-byte MAC addresses and 2 bytes for ethertype). When you specify MTU size on the IOS classic (either with the ip mtu command or with the mtu command), you specify the maximum size of the layer-3 payload without the layer-2 header. Obviously IOS XR works differently – there you have to specify the maximum size of a layer-2 frame, not of its layer-3 payload (comments describing how other platforms behave are most welcome!).

7 comments:

  1. StuckInActive04 July, 2011 07:40

    Junos also includes L2 encaps with MTU. Additionally, if you define your interface with 'flexible-vlan-tagging' you also get another 4 bytes of encaps for the q-in-q tag.

    Luckily, Junos has a handy way of determining the equivalent IOS MTU/L2-less MTU. 'sh interface <name> detail' will print both the "ip mtu" and even the mpls MTU for you.</name>

    ReplyDelete
  2. Russell Heilling04 July, 2011 08:14

    I have been explaining why MTU is a heavily misused term a lot recently. When talking about the raw layer 2 maximum PDU size I prefer to use maximum frame size, as it should be unambiguous. Of course in this case the 1514 bytes is not the maximum frame size, it includes the header + payload but excludes the 4 byte FCS trailer.

    It seems that in this case IOS-XR is acting exactly like JunOS, as the interface MTU is specified exactly this way there too. In JunOS the reason for excluding the FCS is a quirk in the hardware forwarding. I don't know the reasoning in IOS-XR...

    ReplyDelete
  3. Yup, ran into this as well but on the CRS platform, which of course is XR based. Maybe the XR guys are ex JunOS guys? who knows, but seems to be fairly consistent across XR based Cisco devices.

    ReplyDelete
  4. Is there a good reason why the mtu definition is not standardized across platforms ?

    ReplyDelete
  5. I ran into this MTU discussion as well several times, and everytime is getting me an headache.
    Last time I had this discussion on Alcatel Lucent 7750SR Routers.
    The MTU size configured on the port is the layer 2 header and also the layer 3 payload.

    So you have to specify the max size of the layer 2 frame.

    ReplyDelete
  6. On ASR9000 routers, a "show ip interface" tells what's left for IP.
    Example output from "show ip interface te0/0/0/0 | i MTU":
    MTU is 9216 (9202 is available to IP)
    !
    The MTU difference between classic IOS and XR can also hit you when you xconnect between platforms.

    ReplyDelete
  7. The "MTU" should really always be the payload inside the Layer 2 frame, i.e. default 1500. Any other use of "MTU" is in my opinion both incorrect and potential harmful.

    If some device uses the phrase MTU as the maximum Ethernet frame size there are very high risk of confusion and misconfiguration. For example, would "1514" as in the text above still allow the 4 byte FCS checksum and possible a 4 byte 802.1Q tag?

    ReplyDelete

You don't have to log in to post a comment, but please do provide your real name/URL. Anonymous comments might get deleted.

Ivan Pepelnjak, CCIE#1354, is the chief technology advisor for NIL Data Communications. He has been designing and implementing large-scale data communications networks as well as teaching and writing books about advanced technologies since 1990. See his full profile, contact him or follow @ioshints on Twitter.