IBGP or EBGP in an enterprise network?
I got the following question from one of my readers:
I recently started working at a very large enterprise and learnt that the network uses BGP internally. Running IBGP internally is not that unexpected, but after some further inquiry it seems that we are running EBGP internally. I must admit I'm a little surprised about the use of EBGP internally and I wanted to know your thoughts on it.
Although they are part of the same protocol, IBGP and EBGP solve two completely different problems; both of them can be used very successfully in a large enterprise network.
If all you want to do is to scale your network beyond the IGP limits, or make sure the access network flaps don’t affect the core, IBGP is the tool to use. As a downside, every IBGP router must be able to reach every BGP next hop in the same AS (you could change IBGP next hops with inbound route maps, but don’t do that outside of the CCIE lab). The BGP next hops are usually propagated through IGP (propagating them with BGP might even work, but wouldn’t be too far from the adventures of Baron Munchausen); the traffic flow between BGP next hops is thus completely controlled by IGP route selection rules.
EBGP is a completely different beast. Next hops are not propagated across multiple autonomous systems (yet again, there are exceptions, rarely useful beyond the perimeter of a CCIE lab); the router advertising the best EBGP path is generally the one forwarding the traffic into the adjacent AS. You can use this property of EBGP to implement complex routing policies in your network.
In a well-designed network, EBGP sessions usually follow physical layer 3 connectivity; IGBP sessions are usually established between routers that are further apart (commonly between edge routers and a set of centralized route reflectors). IBGP convergence is thus typically faster than EBGP convergence. Even more, IBGP does not need to be involved in the convergence process following a core link or node failure (IGP takes care of that), while you have to rely on EBGP to find alternate paths in EBGP-based networks.
You would run your whole enterprise network as a single AS (using only IBGP) if the only goal you wanted to reach is the increased stability of your core network. A mixture of IBGP and EBGP makes sense if you want to implement routing policies between regions (or countries/continents in a global network) and don’t care too much about the inter-regional convergence time. Using just EBGP with every BGP-speaking router in its own AS probably calls for a network redesign (MPLS/VPN networks where small single-router sites use EBGP to exchange routes with the PE-routers is an obvious exception).
There are many ways to route a large global network with multiple security domains and different teams of network engineers working independently. I can't imagine Google or Microsoft not using eBGP internally. It provides such a very nice clean delineation between different parts of the network. I have do direct knowledge of the MSFT network, so I'll use them as an example, I hope that their Bing and XBox Live and MSDN sites are managed somewhat independently, after all they are all likely pretty big networks with different security requirements and perhaps different protocol requirements. XBox, for example, might want to leverage multicast, whereas Bing might want to optimize for protocol simplicity. When different parts of the network have different business requirements, IMHO, it makes sense to isolate them with eBGP.
Confederations are great whenever one group of engineers wants to divide up their own network into more manageable chunks, but with my example above, do confederations still seem to be the most logical solution? I think ASN scale more with the number of network engineers than routers. Just a SWAG.
Cheers!
Can you update your website so we can share these articles on Twitter or even facebook?
Great article - keep up the good work Ivan!
John
Great post. In fact it was absolutely fabulous.
Would love to see a diagram of what you're talking about to be sure I fully understand (I dont ...quite , right now).
can you please clarify your stament "If you want to scale your network or make sure the access network flaps don’t affect the core, IBGP is the tool to use." ? Is eBGP not as scalable or stable?
Besides limited number of private ASNs needed for a large enterprise network are there any other disadvantages of using eBGP?
Thanks George
EBGP is absolutely stable and scalable (even more so than IBGP), but is slower to converge.
I have discovered that an IBGP route is not being removed from the IP RIB and BGP table when the IBGP neighbor address (which is also the BGP Next Hop) is within the range of an IBGP-learnt supernet route.
This is discovered when a PBR with the "verify-reachability" option is not working as intended upon link failures to the next-hop IBGP neighbor. Indeed, the "recurvise" option for PBR is the best option, but apparently it is a C3750 switch running 12.2SE and the "recurvise" option is not available. :)
This is documented as Scenario #3 in the following blog post.
http://www.itcertnotes.com/2011/08/observation-upon-ibgp-and-ebgp-route.html
Appreciate someone can share some info on whether this behavior is described somewhere in the BGP RFCs, and whether this is Cisco-specific. Thanks. :-)
If the BGP next hop is reachable (even though it's reachable through a BGP route), the BGP route is considered valid. You might want to check the fast fallover and BGP next hop checking features available in Cisco IOS.
http://www.nil.si/ipcorner/DesigningBGPNetworks/
http://www.cisco.com/en/US/docs/ios/12_4t/ip_route/configuration/guide/brbadv.html#wp1109430
I was wondering if it was possible to use IBGP over an MPLS VPN network to interconnect routers on different sites, in many different countries. I am working on an enterprise network which is migrating with two different SPs for the MPLS VPN (not all sites will be eligible to the two SPs) and I don't want to rely on an IGP for this kind of infrastructure. I was thinking about using IBGP between my enterprise routers over the two MPLS VPNs, do you think it is possible ? a good idea ? I'm not sure that EBGP would be a good decision because on every site I would have to manage the two SPs on every site multihomed with a common AS number on each site.
Your point of view ?
Thx
/plg
You should also consider using the ISPs as pure IP transport and run LISP or DMVPN over their IP infrastructures.
BTW, if you'd like to discuss your design with me, this might not be a bad option: http://www.ioshints.info/ExpertExpress
The following configuration is what 95% of all ISPs run and clearly the Chief Network Architect of this global network comes from an ISP.
http://blog.ioshints.info/2011/08/dmvpn-as-backup-for-mplsvpn.html
-IBGP requires the spine router to all be route-reflectors, and a route-map is used on the leaf peering to set the nexthop to the peering address. This essentially makes IBGP function like EBGP.
-To deploy EBGP, the spine is all one AS, and each leaf is a different AS. So every leaf router is its own AS. I found this to consume a lot of private AS space, and operationally be a lot more complicated.
Do you see any other advantages or disadvantages with either design? I think this is a really cool use of BGP in the MSDC's, and wanted to share this since it shows how IBGP can be used without an IGP. :)
Hi Ivan,
This question has always puzzled me. From an enterprise customer preservative, should I run iBGP or iBGP+ IGRP (OSPF/ISIS/EIGRP) or IGP while doing mutual redistribution on the edge routers. I was hoping if you could share some thoughtful insight on when to select one over the another?
Thanks, Khalid