Scaling BGP-Based DMVPN Networks
Cristiano sent me an interesting question:
I saw that to configure BGP as the routing protocol running over DMVPN I have to configure BGP neighbors on the hub site router. Do I really have to configure all the neighbors on the hub site? How many neighbors could I configure? How can I scale that?
According to Cisco Live presentations, BGP-over-DMVPN scales to several thousand spoke sites (per hub router), so you shouldn’t be too worried about the protocol scalability. Configuring all those neighbors is a different issue.
Fortunately anonymous BGP neighbors (which were banned from Cisco IOS years ago) reappeared in a different disguise (dynamic BGP neighbors) in IOS release 15.1T – you can configure an access list which defines whether the remote IP address is a valid BGP neighbor, and accept all incoming TCP requests on port 179 matched by the ACL. The “only” drawback: dynamic neighbors matched by a single bgp listen peer-group command must share the same neighbor parameters, including an AS number.
You can list several AS numbers in the neighbor peer-group remote-as command, but that doesn’t exactly help you if you have 500 remote sites.
Faced with the requirement to have the same AS number on all remote sites you could use one of these two designs:
Running IBGP across DMVPN. While you could make it work, I wouldn’t recommend doing that unless you’re a BGP guru. IBGP was not designed to be used without IGP, and there are simply too many things that can bite you if you’re trying to run IBGP without an underlying IGP. You probably don’t want to run an IGP across a large DMVPN network or you wouldn’t be reading this blog post.
Running EBGP across DMVPN with all spoke sites being in the same autonomous system. This design works great for Phase 1 or Phase 3 DMVPN: spoke sites will ignore routes advertised by other spoke sites (due to BGP loop prevention logic), and a default route advertised by the hub router will take care of the inter-site traffic flow.
Phase 2 DMVPN is trickier; every spoke site should have routes from all other spoke sites (with correct BGP next hops) if you want spoke sites to communicate directly. You can disable BGP loop prevention logic on spoke sites (using neighbor allowas-in) or play dirty tricks with the AS path like neighbor remove-private-as or neighbor as-override on the hub router (not recommended).
- BGP routing in DMVPN Access Networks case study;
- DMVPN webinars: buy them individually, or in a bundle.
In case of eBGP you wrote:
>You can disable BGP loop prevention logic on spoke sites (using neighbor allowas-in) or play dirty tricks with the AS path like neighbor remove-private-as or neighbor as-override on the hub router (not recommended).
But what would be the design, if we also have additional (MPLS ? cloud) with eBGP?
(in my case customer wants to be able to prefer on path over another - based on community - for different spokes, i.e. spoke 1 is preffered over DMVPN, spoke2 - over MPLS).
So, how would you suggest to configure DMVPN (and dynamic peering), keeping loop-prevention logic in place?
If you need more, you can always engage me for online consulting
Given that a single DMVPN functions as one IP subnet, what sorts of things can bite you if you use iBGP? Perhaps I'm not thinking it through completely, but with all hubs and spokes being (technically) directly connected, how would an IGP make any difference to iBGP operation in this case?
Also you mention that with eBGPoDMVPN, all the spoke sites are placed in the same autonomous system, just wondering how's that possible if all the spokes are in separate locations across different providers (unless we are using something like GRE to build that iBGP mesh among all the spoke sites). If indeed that's the answer, would it not be cleaner to have different AS for each spoke's site (say using the 32 bit AS notation)?
Using dynamic neighbor command, and suitable crypto design, all IBGP neighbors are on same subnet and there is no per-spoke config on the hub. Hub and spokes IBGP peer using Physical DMVPN subnet (update-source TunnelX), so there is no IGP to spokes needed. Spokes only peer with Hub. Hub acts as route-reflector, it all works. Next-hop-self all option gives the option to control whether you are pure-hub and spoke or to allow spoke-to-spoke routing via dynamic tunnel. In our case there is minimal/no spoke-to-spoke so we can set "all", and/or route-map the hub-spoke peerings to only send default.
In thinking about how we might convert to using dynamic neighbors, it occurred to me that rather than play games with loop prevention, why not simply select an unused AS to represent the DMVPN cloud and apply the appropriate local-as command to the DMVPN BGP peers? Each site remains a separate AS, AS paths correctly reflect the true origin AS, and loop prevention should still work properly, but the DMVPN hub sees all spoke sessions as coming from the common phantom AS, which should allow dynamic neighbors to work just fine.
How to achieve direct spoke-to-spoke communication without hub in the middle?
You'll find more details in other posts on this blog (just search for DMVPN) or in my DMVPN webinars.
Awesome post as usual. DMVPN is one of my favorite technologies to mess around with. This is not important in the grand scheme of the focal point of the post but i thought to notify you of a typo: the line 'and accept all incoming TCP requests on port 79 matched by the ACL' should read port 179 (BGP).