Scaling BGP-Based DMVPN Networks

Cristiano sent me an interesting question:

I saw that to configure BGP as the routing protocol running over DMVPN I have to configure BGP neighbors on the hub site router. Do I really have to configure all the neighbors on the hub site? How many neighbors could I configure? How can I scale that?

According to Cisco Live presentations, BGP-over-DMVPN scales to several thousand spoke sites (per hub router), so you shouldn’t be too worried about the protocol scalability. Configuring all those neighbors is a different issue.

Fortunately anonymous BGP neighbors (which were banned from Cisco IOS years ago) reappeared in a different disguise (dynamic BGP neighbors) in IOS release 15.1T – you can configure an access list which defines whether the remote IP address is a valid BGP neighbor, and accept all incoming TCP requests on port 79 matched by the ACL. The “only” drawback: dynamic neighbors matched by a single bgp listen peer-group command must share the same neighbor parameters, including an AS number.

You can list several AS numbers in the neighbor peer-group remote-as command, but that doesn’t exactly help you if you have 500 remote sites.

Faced with the requirement to have the same AS number on all remote sites you could use one of these two designs:

Running IBGP across DMVPN. While you could make it work, I wouldn’t recommend doing that unless you’re a BGP guru. IBGP was not designed to be used without IGP, and there are simply too many things that can bite you if you’re trying to run IBGP without an underlying IGP. You probably don’t want to run an IGP across a large DMVPN network or you wouldn’t be reading this blog post.

Running EBGP across DMVPN with all spoke sites being in the same autonomous system. This design works great for Phase 1 or Phase 3 DMVPN: spoke sites will ignore routes advertised by other spoke sites (due to BGP loop prevention logic), and a default route advertised by the hub router will take care of the inter-site traffic flow.

Phase 2 DMVPN is trickier; every spoke site should have routes from all other spoke sites (with correct BGP next hops) if you want spoke sites to communicate directly. You can disable BGP loop prevention logic on spoke sites (using neighbor allowas-in) or play dirty tricks with the AS path like neighbor remove-private-as or neighbor as-override on the hub router (not recommended).

More information

12 comments:

  1. Hello, Ivan.

    In case of eBGP you wrote:
    >You can disable BGP loop prevention logic on spoke sites (using neighbor allowas-in) or play dirty tricks with the AS path like neighbor remove-private-as or neighbor as-override on the hub router (not recommended).

    But what would be the design, if we also have additional (MPLS ? cloud) with eBGP?
    (in my case customer wants to be able to prefer on path over another - based on community - for different spokes, i.e. spoke 1 is preffered over DMVPN, spoke2 - over MPLS).
    So, how would you suggest to configure DMVPN (and dynamic peering), keeping loop-prevention logic in place?

    ReplyDelete
    Replies
    1. Some of those aspects are described here:

      http://www.ipspace.net/Integrating_Internet_VPN_with_MPLS_VPN_WAN

      If you need more, you can always engage me for online consulting

      http://www.ipspace.net/ExpertExpress

      Delete
  2. Hey Ivan:

    Given that a single DMVPN functions as one IP subnet, what sorts of things can bite you if you use iBGP? Perhaps I'm not thinking it through completely, but with all hubs and spokes being (technically) directly connected, how would an IGP make any difference to iBGP operation in this case?

    Jody

    ReplyDelete
  3. If the DMVPN subnet is the only place you run BGP in your network (probably not), or if it uses a different AS number (which would have to be used on the hub router), you're OK. Otherwise you'll need route reflectors somewhere (hub routers are the obvious choice) … and hit the problem of not being able to modify BGP next hop on reflected routes.

    ReplyDelete
    Replies
    1. The problem being that the reflected next-hop attribute is not known within the IGP domain if I understand correctly.

      Delete
    2. More precisely, the problem is the lack of IGP domain (and thus information about non-DMVPN next hops) in the DMVPN subnet.

      Delete
  4. I think for an iBGP DMVPN design the BGP Dynamic Neighbors works well but not necessarily that useful for a eBGP DMVPN because your eBGP neighbors most likely won't fall into a certain range which will render the feature much less effective.

    Also you mention that with eBGPoDMVPN, all the spoke sites are placed in the same autonomous system, just wondering how's that possible if all the spokes are in separate locations across different providers (unless we are using something like GRE to build that iBGP mesh among all the spoke sites). If indeed that's the answer, would it not be cleaner to have different AS for each spoke's site (say using the 32 bit AS notation)?

    ReplyDelete
    Replies
    1. EBGP is run across DMVPN tunnel, not with the ISP, so the AS numbers don't matter - ISPs usually don't see them.

      Delete
    2. That makes sense, but if all the spokes sites are in the same AS, then do all the spokes still need to be in the same iBGP full mesh? (I can see the benefit with the dynamic neighbor with that design though).

      Delete
  5. read this with interest.. have built a network such as this for a large enterprise. IBGP works just fine , and I consider it easier operationally and would recommend it over EBGP (having a separate AS for each spoke surely is very painful?).
    Using dynamic neighbor command, and suitable crypto design, all IBGP neighbors are on same subnet and there is no per-spoke config on the hub. Hub and spokes IBGP peer using Physical DMVPN subnet (update-source TunnelX), so there is no IGP to spokes needed. Spokes only peer with Hub. Hub acts as route-reflector, it all works. Next-hop-self all option gives the option to control whether you are pure-hub and spoke or to allow spoke-to-spoke routing via dynamic tunnel. In our case there is minimal/no spoke-to-spoke so we can set "all", and/or route-map the hub-spoke peerings to only send default.

    ReplyDelete
  6. We have a MPLS/VPN solution from a service provider, with a DMVPN over the Internet for backup. We currently enforce the use of a unique AS per site to ease troubleshooting of routing problems in the SP network, and currently have our DMVPN headend routers configured with static BGP neighbors.

    In thinking about how we might convert to using dynamic neighbors, it occurred to me that rather than play games with loop prevention, why not simply select an unused AS to represent the DMVPN cloud and apply the appropriate local-as command to the DMVPN BGP peers? Each site remains a separate AS, AS paths correctly reflect the true origin AS, and loop prevention should still work properly, but the DMVPN hub sees all spoke sessions as coming from the common phantom AS, which should allow dynamic neighbors to work just fine.

    ReplyDelete
    Replies
    1. Interesting concept. Should work just fine, and you'll retain the "site AS" in the AS path. Thank you!

      Delete

You don't have to log in to post a comment, but please do provide your real name/URL. Anonymous comments might get deleted.

Ivan Pepelnjak, CCIE#1354, is the chief technology advisor for NIL Data Communications. He has been designing and implementing large-scale data communications networks as well as teaching and writing books about advanced technologies since 1990. See his full profile, contact him or follow @ioshints on Twitter.