After we get rid of the QoS FUD, the next question I usually get when discussing overlay networks is “how should these networks treat IP TTL?”

As (almost) always, the answer is “It depends.”

Layer-2 Virtual Networks

Overlay virtual networking solutions like VXLAN that implement layer-2 segments (effectively Ethernet-over-something) should not modify the VM-generated traffic. These solutions are emulating a transparent bridge and should NOT interact with the user traffic; all they can do is forward, flood or drop.

Obviously the transport TTL (TTL generated by hypervisor when encapsulating the VM traffic) shouldn’t reflect the VM-generated TTL. VM-generated TTL could be anything (VM could also generate non-IP traffic), while the transport TTL needs to be high enough to allow the packet to traverse the data center core.

Conclusions:

• Don’t touch the overlay (VM) TTL;
• Use whatever TTL makes sense in the transport network.

Layer-3 Virtual Networks

Solutions that implement layer-3 forwarding are usually emulating Ethernet segments (layer-2 segments) connected with routers. In some cases the whole virtual network acts as a single virtual router (VMware NSX Distributed Router, Hyper-V, NEC ProgrammableFlow …), in others the inter-subnet traffic flows through a gateway appliance or a VM (VMware NSX Services Router, default OpenStack networking …).

These solutions SHOULD decrement TTL like any other router (or layer-3 switch) would do. If they wish to stay as close to the emulated Ethernet behavior as possible, they SHOULD decrement TTL if and only if the packet crosses subnet boundaries (or you might get crazy problems with application software that sends packets with TTL = 1).

For example, Hyper-V Network Virtualization SHOULD NOT decrement TTL if the source and destination VM belong to the same subnet (even though the HNV module actually performs L3 lookup to figure out where to send the packet) but SHOULD decrement TTL if the destination VM belongs to another IPv4 or IPv6 subnet.

Like in the layer-2 case, the transport TTL has nothing in common with the VM-generated TTL – hypervisors should use whatever TTL they need to get the encapsulated traffic across the data center fabric.

Conclusions:

• Decrement TTL like a router would do;
• Don’t copy overlay TTL into transport TTL or vice versa;
• Use whatever TTL makes sense in the transport network.

But this is not how MPLS works

Really? Well, this is EXACTLY how L2VPNs (EoMPLS, VPLS, EVPN) work.

MPLS-based L3VPN (the “original” MPLS/VPN) is a totally different story: it’s not supposed to emulate a single virtual router, but a whole WAN. Copying customer TTL into provider TTL (and vice versa) is the most natural thing to do under those circumstances (unless the MPLS provider disables TTL propagation because they want to hide the internal network details).

More information

Watch the Cloud Computing Networking webinar if you need an overview of various virtual network technologies, Overlay Virtual Networking for an overview of what major vendors have to offer, VXLAN deep dive if you’re interested in VXLAN implementation details and VMware NSX Technical Deep DiveArchitecture if you want to know how NSX works.