Scaling BGP-Based DMVPN Networks

Cristiano sent me an interesting question:

I saw that to configure BGP as the routing protocol running over DMVPN I have to configure BGP neighbors on the hub site router. Do I really have to configure all the neighbors on the hub site? How many neighbors could I configure? How can I scale that?

According to Cisco Live presentations, BGP-over-DMVPN scales to several thousand spoke sites (per hub router), so you shouldn’t be too worried about the protocol scalability. Configuring all those neighbors is a different issue.

Fortunately, anonymous BGP neighbors (which were banned from Cisco IOS years ago) reappeared in a different disguise (dynamic BGP neighbors) in IOS release 15.1T – you can configure an access list which defines whether the remote IP address is a valid BGP neighbor, and accept all incoming TCP requests on port 179 matched by the ACL. The “only” drawback: dynamic neighbors matched by a single bgp listen peer-group command must share the same neighbor parameters, including an AS number.

You can list several AS numbers in the neighbor peer-group remote-as command, but that doesn’t exactly help you if you have 500 remote sites.

Faced with the requirement to have the same AS number on all remote sites you could use one of these two designs:

Running IBGP across DMVPN. While you could make it work (even easier now that Cisco tweaked its BGP route reflector behavior), I wouldn’t recommend doing that unless you’re a BGP guru. IBGP was not designed to be used without IGP, and there are simply too many things that can bite you if you’re trying to run IBGP without an underlying IGP. You probably don’t want to run an IGP across a large DMVPN network or you wouldn’t be reading this blog post.

Running EBGP across DMVPN with all spoke sites being in the same autonomous system. This design works great for Phase 1 or Phase 3 DMVPN: spoke sites will ignore routes advertised by other spoke sites (due to BGP loop prevention logic), and a default route advertised by the hub router will take care of the inter-site traffic flow.

Phase 2 DMVPN is trickier; every spoke site should have routes from all other spoke sites (with correct BGP next hops) if you want spoke sites to communicate directly. You can disable BGP loop prevention logic on spoke sites (using neighbor allowas-in) or play dirty tricks with the AS path like neighbor remove-private-as or neighbor as-override on the hub router (not recommended).

More Information

Recent posts in the same categories

21 comments:

  1. Anonymous 17 March 2014 10:56
    Hello, Ivan.

    In case of eBGP you wrote:
    >You can disable BGP loop prevention logic on spoke sites (using neighbor allowas-in) or play dirty tricks with the AS path like neighbor remove-private-as or neighbor as-override on the hub router (not recommended).

    But what would be the design, if we also have additional (MPLS ? cloud) with eBGP?
    (in my case customer wants to be able to prefer on path over another - based on community - for different spokes, i.e. spoke 1 is preffered over DMVPN, spoke2 - over MPLS).
    So, how would you suggest to configure DMVPN (and dynamic peering), keeping loop-prevention logic in place?
    Replies
    1. Ivan Pepelnjak 17 March 2014 16:01
      Some of those aspects are described here:

      http://www.ipspace.net/Integrating_Internet_VPN_with_MPLS_VPN_WAN

      If you need more, you can always engage me for online consulting

      http://www.ipspace.net/ExpertExpress
  2. Unknown 17 March 2014 20:08
    Hey Ivan:

    Given that a single DMVPN functions as one IP subnet, what sorts of things can bite you if you use iBGP? Perhaps I'm not thinking it through completely, but with all hubs and spokes being (technically) directly connected, how would an IGP make any difference to iBGP operation in this case?

    Jody
  3. Ivan Pepelnjak 18 March 2014 20:21
    If the DMVPN subnet is the only place you run BGP in your network (probably not), or if it uses a different AS number (which would have to be used on the hub router), you're OK. Otherwise you'll need route reflectors somewhere (hub routers are the obvious choice) … and hit the problem of not being able to modify BGP next hop on reflected routes.
    Replies
    1. Jeff Behrns 19 March 2014 05:42
      The problem being that the reflected next-hop attribute is not known within the IGP domain if I understand correctly.
    2. Ivan Pepelnjak 19 March 2014 07:27
      More precisely, the problem is the lack of IGP domain (and thus information about non-DMVPN next hops) in the DMVPN subnet.
  4. Unknown 18 March 2014 22:36
    I think for an iBGP DMVPN design the BGP Dynamic Neighbors works well but not necessarily that useful for a eBGP DMVPN because your eBGP neighbors most likely won't fall into a certain range which will render the feature much less effective.

    Also you mention that with eBGPoDMVPN, all the spoke sites are placed in the same autonomous system, just wondering how's that possible if all the spokes are in separate locations across different providers (unless we are using something like GRE to build that iBGP mesh among all the spoke sites). If indeed that's the answer, would it not be cleaner to have different AS for each spoke's site (say using the 32 bit AS notation)?
    Replies
    1. Ivan Pepelnjak 19 March 2014 07:28
      EBGP is run across DMVPN tunnel, not with the ISP, so the AS numbers don't matter - ISPs usually don't see them.
    2. Unknown 19 March 2014 15:50
      That makes sense, but if all the spokes sites are in the same AS, then do all the spokes still need to be in the same iBGP full mesh? (I can see the benefit with the dynamic neighbor with that design though).
  5. Unknown 27 March 2014 17:57
    read this with interest.. have built a network such as this for a large enterprise. IBGP works just fine , and I consider it easier operationally and would recommend it over EBGP (having a separate AS for each spoke surely is very painful?).
    Using dynamic neighbor command, and suitable crypto design, all IBGP neighbors are on same subnet and there is no per-spoke config on the hub. Hub and spokes IBGP peer using Physical DMVPN subnet (update-source TunnelX), so there is no IGP to spokes needed. Spokes only peer with Hub. Hub acts as route-reflector, it all works. Next-hop-self all option gives the option to control whether you are pure-hub and spoke or to allow spoke-to-spoke routing via dynamic tunnel. In our case there is minimal/no spoke-to-spoke so we can set "all", and/or route-map the hub-spoke peerings to only send default.
  6. Phil 07 July 2014 00:36
    We have a MPLS/VPN solution from a service provider, with a DMVPN over the Internet for backup. We currently enforce the use of a unique AS per site to ease troubleshooting of routing problems in the SP network, and currently have our DMVPN headend routers configured with static BGP neighbors.

    In thinking about how we might convert to using dynamic neighbors, it occurred to me that rather than play games with loop prevention, why not simply select an unused AS to represent the DMVPN cloud and apply the appropriate local-as command to the DMVPN BGP peers? Each site remains a separate AS, AS paths correctly reflect the true origin AS, and loop prevention should still work properly, but the DMVPN hub sees all spoke sessions as coming from the common phantom AS, which should allow dynamic neighbors to work just fine.
    Replies
    1. Ivan Pepelnjak 07 July 2014 07:31
      Interesting concept. Should work just fine, and you'll retain the "site AS" in the AS path. Thank you!
    2. Anonymous 03 March 2015 03:40
      Phil - Did this work out? I had this same idea, but am cautious about being able to use route-maps and community filters on the RR neighbor. Any thoughts or updates?
  7. David 08 February 2015 14:04
    I am testing out Phil's suggestion for a hub and spoke WAN design. Nice, tidy configuration, but so far the only way I have found to propagate routes between spokes is by using as-override at the hub. The path at spoke2 appears as "AS_HUB AS_HUB AS_SPOKE1" removing the phantom AS. Is there a better way?
    Replies
    1. Ivan Pepelnjak 08 February 2015 14:07
      If you want to cheat the basic BGP rules you'll increase the complexity of your setup one way or another. It's either allowas-in on spokes or as-override on hub.
  8. David 08 February 2015 17:57
    Thanks, Ivan. Static neighbors it is.

  9. Anonymous 05 April 2015 18:12
    What about DMVPN cloud with external BGP running on it. Spoke-to-spoke communication is currently traversing the hub. All spokes are using different AS numbers.
    How to achieve direct spoke-to-spoke communication without hub in the middle?
  10. Ivan Pepelnjak 06 April 2015 09:06
    BGP next-hop handling should solve that problem automatically (assuming you're not using next-hop-self on the hub), or you could go for DMVPN phase 3.

    You'll find more details in other posts on this blog (just search for DMVPN) or in my DMVPN webinars.
  11. Anonymous 06 April 2015 21:12
    Thank you, Ivan.

    Regards,
    Nikolay
  12. Unknown 25 June 2015 13:40
    Hello Ivan,

    Awesome post as usual. DMVPN is one of my favorite technologies to mess around with. This is not important in the grand scheme of the focal point of the post but i thought to notify you of a typo: the line 'and accept all incoming TCP requests on port 79 matched by the ACL' should read port 179 (BGP).

    Regards.
    Replies
Add comment
Home
Sidebar