Redundant DMVPN designs, Part 2 (Multiple Uplinks)

In the Redundant DMVPN Design, Part 1 I described the options you have when you want to connect non-redundant spokes to more than one hub. In this article, we’ll go a step further and design hub and spoke sites with multiple uplinks.

Public IP addressing

Fact: DMVPN tunnel endpoints have to use public IP addresses or the hub/spoke routers wouldn’t be able to send GRE/IPsec packets across the public backbone.

The public IP address in this context is an IP address routable across the Service Provider backbone your routers are connected to. If you run DMVPN over the Internet, then the public IP address is a globally routable IPv4 or IPv6 address; if you run DMVPN over an MPLS/VPN infrastructure, then the public IP address is whatever IP address is routable across the MPLS/VPN network (it could still be an RFC 1918 IPv4 address).

Public IP addresses from the service provider PA address space usually get assigned to individual uplinks on the spoke routers. These IP addresses could be static or dynamically allocated through mechanisms like DHCP/DHCPv6, SLAAC or IPCP.

SMB customers not having their own IP address space would most often deploy hub routers with public IP addresses assigned to individual links (similar to spoke sites). Larger organizations with their own IP address space and AS number would use BGP with the upstream service providers and a single IP address (from their IP address space) on the hub router:

DMVPN tunnels on spoke sites

Fact: DMVPN tunnel endpoints are tied to public IP address and thus to individual uplinks. Uplink failure causes IP address loss and subsequently tunnel interface failure.

Conclusion: If you want to maximize the spoke site redundancy, you have to deploy a separate DMVPN tunnel interface on each spoke router uplink.

Fact: Cisco IOS doesn’t allow the same IP subnet to be used on multiple mGRE interfaces.

Conclusion#1: Multiple DMVPN tunnel interfaces deployed on a spoke router have to belong to different IP subnets.

Conclusion#2: You need multiple DMVPN tunnels in your network. The number of DMVPN interfaces on hub routers depends on your resiliency requirements (see below).

Outbound routing on spoke routers

You can never be sure whether the upstream ISPs use RPF filters or other filtering mechanisms. It’s thus mandatory to ensure proper outbound packet forwarding over the spoke uplinks: tunnel interface associated with IP address belonging to ISP-A can send packets only over the uplink connecting the spoke router to ISP-A.

You can use tunnel route-via feature to force the tunnel interfaces to use correct uplinks (but it doesn’t work if you use IPsec), or you could use VRF-based tunnel interfaces:

  • Create multiple VRFs on the spoke router, one per uplink;
  • Create a static default route in each VRF pointing to the associated uplink;
  • Configure transport VRF on each DMVPN tunnel interface.

If you’re not familiar with VRFs, use the tested router configurations you get with the DMVPN – from basics to scalable networks webinar.

DMVPN tunnels on hub sites

The number of DMVPN tunnels on the hub sites depends on the DMVPN model you’re using (Phase 1/2/3) and the redundancy requirements.

Assuming a two-hub network has to survive a single failure (hub router or spoke uplink), use a single hub per DMVPN tunnel (unless you're using Phase 3 DMVPN, in which case all hub routers probably have to be connected to all tunnels).

Spoke uplink failure will not isolate a spoke (it still has the other uplink), hub router failure will also not break the network (spokes can still communicate through the other hub). However, a worst-case combination of hub router and spoke uplink failure might isolate a spoke site.

If your network has to be more failure-resilient, connect all hubs to all DMVPN tunnels. This design will be able to survive multiple failures (including a concurrent hub router and spoke uplink failure).

Finally, if you’re using DMVPN Phase 2, you might need to deploy even more tunnels to avoid the NHRP convergence problems. In our sample network you would need two DMVPN tunnels on each hub router (one for ISP-A spoke uplinks, the other one for ISP-B spoke uplinks) and four tunnels on the spoke routers (one for each uplink/hub combination).

Alternate solution: VRF fiesta

KAV proposed an alternate solution in a comment to the Redundant DMVPN Design, Part 1 post: put each DMVPN tunnel interface in a separate VRF and use redistribution through BGP in a VRF-lite environment to collect routes from both DMVPN VRFs in the LAN-side VRF.

Advantage: since each DMVPN tunnel belongs to a different VRF, you can use the same IP subnet on all spoke-router tunnels, minimizing the overall number of DMVPN tunnels (you can use a single DMVPN subnet if you’re running Phase 1 or Phase 3 DMVPN).

Drawback: you need between three and five VRFs (three to support this design, two more for split default routing in Phase 2/3 DMVPN) on the spoke routers and route redistribution through BGP. Guess whose phone will ring when a spoke router experiences “mission-critical” problems at 2AM.

Need help?

Start with the DMVPN – from basics to scalable networks webinar; it probably contains 95% of what you need to know about DMVPN (the DMVPN New Features one describes DMVPN features introduced in IOS releases 15.x). The Enterprise MPLS/VPN Deployment webinar describes everything you ever wanted to know about VRFs, VRF-Lite and tunnel interfaces running inside and over VRFs. You get access to all three webinars with the yearly subscription.

If you need a second opinion or help with your DMVPN design, check out the ExpertExpress service. You could also engage our professional services team for larger design/deployment projects.

7 comments:

  1. comment to "Drawback: you need between three and five VRFs (three to support this design, two more for split default routing in Phase 2/3 DMVPN) on the spoke routers and route redistribution through BGP."

    in our configuration we need to redistribute about 2 000 routes for different reasons and this configuration also working on CISCO871 routers with IOS version adventerprisek9-mz.124-24.T and higher without problems. the main thing - properly configure routing.

    ReplyDelete
  2. I'm not saying it wouldn't work or couldn't be done. Actually, if I would do the design a few years ago, I would probably end up with something similarly elegant (because it is an elegant solution).

    However, I wouldn't want to see what happens when a junior CCNP-level engineer has to troubleshoot this beast during the graveyard shift.

    ReplyDelete
  3. Ivan, I like the quick-and-dirty, back-of-the-napkin diagrams. Are you using a drawing program on the iPad to make those? The output looks very similar to Penultimate which I've also used.

    ReplyDelete
  4. Yes, it is Penultimate.

    ReplyDelete
  5. Hi I have Some question a bout dmvpn design
    We have about 200 branch with cisco 1841 router and above (2811, 3848) and We have 2 WAN SP that provide 2 layer 3 VPN Service (MPLS VPN Layer 3 and WiMax) MPLS network is full mesh in nature but WiMax isn't and every branch have both of this link to HQ router in Centeral Site we have 2 3945 router each one for one ISP.
    I plan to provide HA in this network and upgrade dmvpn over mpls network to phase 3 to support Voip ... please give me any advise for this project.
    sorry for bad English

    ReplyDelete
  6. and please introduce some reference for dmvpn desing for me!!
    I cant access to your resource because of money and etc.

    ReplyDelete
    Replies
    1. Let me rephrase your question: you have a 200+ branch network connected to 2 service providers and yet don't have the funding to solve your design problems?

      Delete

You don't have to log in to post a comment, but please do provide your real name/URL. Anonymous comments might get deleted.

Ivan Pepelnjak, CCIE#1354, is the chief technology advisor for NIL Data Communications. He has been designing and implementing large-scale data communications networks as well as teaching and writing books about advanced technologies since 1990. See his full profile, contact him or follow @ioshints on Twitter.