IBGP or EBGP in an enterprise network?

I got the following question from one of my readers:

I recently started working at a very large enterprise and learnt that the network uses BGP internally. Running IBGP internally is not that unexpected, but after some further inquiry it seems that we are running EBGP internally. I must admit I'm a little surprised about the use of EBGP internally and I wanted to know your thoughts on it.

Although they are part of the same protocol, IBGP and EBGP solve two completely different problems; both of them can be used very successfully in a large enterprise network.

If all you want to do is to scale your network beyond the IGP limits, or make sure the access network flaps don’t affect the core, IBGP is the tool to use. As a downside, every IBGP router must be able to reach every BGP next hop in the same AS (you could change IBGP next hops with inbound route maps, but don’t do that outside of the CCIE lab). The BGP next hops are usually propagated through IGP (propagating them with BGP might even work, but wouldn’t be too far from the adventures of Baron Munchausen); the traffic flow between BGP next hops is thus completely controlled by IGP route selection rules.

EBGP is a completely different beast. Next hops are not propagated across multiple autonomous systems (yet again, there are exceptions, rarely useful beyond the perimeter of a CCIE lab); the router advertising the best EBGP path is generally the one forwarding the traffic into the adjacent AS. You can use this property of EBGP to implement complex routing policies in your network.

In a well-designed network, EBGP sessions usually follow physical layer 3 connectivity; IGBP sessions are usually established between routers that are further apart (commonly between edge routers and a set of centralized route reflectors). IBGP convergence is thus typically faster than EBGP convergence. Even more, IBGP does not need to be involved in the convergence process following a core link or node failure (IGP takes care of that), while you have to rely on EBGP to find alternate paths in EBGP-based networks.

You would run your whole enterprise network as a single AS (using only IBGP) if the only goal you wanted to reach is the increased stability of your core network. A mixture of IBGP and EBGP makes sense if you want to implement routing policies between regions (or countries/continents in a global network) and don’t care too much about the inter-regional convergence time. Using just EBGP with every BGP-speaking router in its own AS probably calls for a network redesign (MPLS/VPN networks where small single-router sites use EBGP to exchange routes with the PE-routers is an obvious exception).

20 comments:

  1. Russell Heilling24 August, 2011 10:38

    Another reason why EBGP may be seen inside an enterprise network would the use of a carrier provided MPLS L3VPN, as only EBGP is supported when BGP is used at the PE-CE interface. Support for IBGP is in the works, but currently still at draft stage: http://tools.ietf.org/html/draft-ietf-l3vpn-ibgp-08

    ReplyDelete
  2. You might use ibgp with confederations, which would feel more the right way to do it, rather than doing ebgp with multiple AS.

    ReplyDelete
  3. Peter John Hill24 August, 2011 12:24

    After a few months, it might not be as surprising as it is now for the reader. I'd much rather use eBGP to craft route policy than any IGP. IGP's are great for fast convergence and finding the shortest path, but there is a reason we don't use it on the Internet.

    There are many ways to route a large global network with multiple security domains and different teams of network engineers working independently. I can't imagine Google or Microsoft not using eBGP internally. It provides such a very nice clean delineation between different parts of the network. I have do direct knowledge of the MSFT network, so I'll use them as an example, I hope that their Bing and XBox Live and MSDN sites are managed somewhat independently, after all they are all likely pretty big networks with different security requirements and perhaps different protocol requirements. XBox, for example, might want to leverage multicast, whereas Bing might want to optimize for protocol simplicity. When different parts of the network have different business requirements, IMHO, it makes sense to isolate them with eBGP.

    Confederations are great whenever one group of engineers wants to divide up their own network into more manageable chunks, but with my example above, do confederations still seem to be the most logical solution? I think ASN scale more with the number of network engineers than routers. Just a SWAG.

    Cheers!

    ReplyDelete
  4. Hi,

    Can you update your website so we can share these articles on Twitter or even facebook?

    ReplyDelete
  5. Ivan Pepelnjak24 August, 2011 15:12

    How about clicking the "facebook" or "twitter" button just above my bio?

    ReplyDelete
  6. It's also important to consider whether IBGP or EBGP will integrate with other services (like MPLS/VPLS). Some companies I know use EIGRP in combination with VPLS. They are currently considering moving to EBGP though.

    Great article - keep up the good work Ivan!

    John

    ReplyDelete
  7. Ivan,

    Great post. In fact it was absolutely fabulous.

    Would love to see a diagram of what you're talking about to be sure I fully understand (I dont ...quite , right now).

    ReplyDelete
  8. Ivan,
    can you please clarify your stament "If you want to scale your network or make sure the access network flaps don’t affect the core, IBGP is the tool to use." ? Is eBGP not as scalable or stable?
    Besides limited number of private ASNs needed for a large enterprise network are there any other disadvantages of using eBGP?
    Thanks George

    ReplyDelete
  9. Ivan Pepelnjak26 August, 2011 19:19

    I probably should have said "If you _only_ want to scale your network ... use IBGP". Reworded. Does it make more sense now?

    EBGP is absolutely stable and scalable (even more so than IBGP), but is slower to converge.

    ReplyDelete
  10. I believe that using modifier "absolutely" is not entirely accurate :) eBGP is /relatively/ scalable, but alas still the only deployed inter-AS routing protocol. Stability at large-scale deployments could be questioned, especially if it's not explicitly defined - e.g. Internet routing tables are never entirely stable and hence global routing using BGP never converges. Scalability is also tricky, and this problem has been intensely discussed for the last 6-7 years at least. While I'm 100% sure you are perfectly familiar with both of these facts, I still wanted to point out that BGP has some well-known drawback and could not serve as a "bulletproof" solution. There are many common issues associated with BGP deployments that are not very well documented, for some reason :)

    ReplyDelete
  11. Peter, it is interested that you selected MSFT network as your example :) I'm not sure if I can talk about it openly, but the design follows many "well-known" templates with some quirks, providing some level of isolation, as you mentioned. Management is another tricky issue, and I believe it is the major scaling factor for large network - probably even more important than platform and protocol limitations. It is interesting that on the networks of that scale, implementation and operations play much more significant role than architectures, in my opinion. After all, designing an architecture for a large network is not that hard. Implementing, operating and maintaing it in good posture is much more challenging problem :)

    ReplyDelete
  12. Ivan Pepelnjak27 August, 2011 11:54

    Would you be happy with "EBGP is like democracy" - not perfect, but still the best option available? :-E

    ReplyDelete
  13. Yap Chin Hoong29 August, 2011 05:29

    Hi all, I would like to take this opportunity to seek some info on a scenario I discovered recently.

    I have discovered that an IBGP route is not being removed from the IP RIB and BGP table when the IBGP neighbor address (which is also the BGP Next Hop) is within the range of an IBGP-learnt supernet route.
    This is discovered when a PBR with the "verify-reachability" option is not working as intended upon link failures to the next-hop IBGP neighbor. Indeed, the "recurvise" option for PBR is the best option, but apparently it is a C3750 switch running 12.2SE and the "recurvise" option is not available. :)

    This is documented as Scenario #3 in the following blog post.
    http://www.itcertnotes.com/2011/08/observation-upon-ibgp-and-ebgp-route.html

    Appreciate someone can share some info on whether this behavior is described somewhere in the BGP RFCs, and whether this is Cisco-specific. Thanks. :-)

    ReplyDelete
  14. Ivan Pepelnjak29 August, 2011 07:17

    It's a FAD (functions-as-designed) 8-)

    If the BGP next hop is reachable (even though it's reachable through a BGP route), the BGP route is considered valid. You might want to check the fast fallover and BGP next hop checking features available in Cisco IOS.

    http://www.nil.si/ipcorner/DesigningBGPNetworks/
    http://www.cisco.com/en/US/docs/ios/12_4t/ip_route/configuration/guide/brbadv.html#wp1109430

    ReplyDelete
  15. Yap Chin Hoong29 August, 2011 07:49

    Thanks Ivan for the fast response! Will look into the links later... :)

    ReplyDelete
  16. Pierre-Louis Gingembre29 August, 2011 15:46

    Hi all,

    I was wondering if it was possible to use IBGP over an MPLS VPN network to interconnect routers on different sites, in many different countries. I am working on an enterprise network which is migrating with two different SPs for the MPLS VPN (not all sites will be eligible to the two SPs) and I don't want to rely on an IGP for this kind of infrastructure. I was thinking about using IBGP between my enterprise routers over the two MPLS VPNs, do you think it is possible ? a good idea ? I'm not sure that EBGP would be a good decision because on every site I would have to manage the two SPs on every site multihomed with a common AS number on each site.

    Your point of view ?

    Thx
    /plg

    ReplyDelete
  17. Ivan Pepelnjak29 August, 2011 18:29

    I wouldn't try to make it work with IBGP, unless both service providers offer Carrier's carrier service. Once I've been involved in a similar design using IBGP and it was "somewhat" nasty.

    You should also consider using the ISPs as pure IP transport and run LISP or DMVPN over their IP infrastructures.

    BTW, if you'd like to discuss your design with me, this might not be a bad option: http://www.ioshints.info/ExpertExpress

    ReplyDelete
  18. Makes sense though we don't have all the details. I assume that this is cBGP or confederated BGP as we used to call it in the old days. Question though, point to point interface and router loopbacks (or equiv) require an IGP like OSPF or ISIS since BGP requires an IGP to for finding next-hop information or routes are inaccessible. Statics are possible, but sloppy.

    The following configuration is what 95% of all ISPs run and clearly the Chief Network Architect of this global network comes from an ISP.

    ReplyDelete
  19. If the VPN is purchased from a major carrier like BT, AT&T or Verizon (RFC2547bis MPLS), then yes, BGP or MP-BGP is always used on the carriers network to supply the service. The IGP that you use is your choice, however, an EGP like BGP is not recommended since it's "heavy" and unnecessary. OSPF/EIGRP are the better choices.

    ReplyDelete
  20. Depends on what they're doing. If they redistribute BGP into IGP on the PE-router then using IGP might make sense; if they have their own CE-routers and redistribute BGP-into-IGP on the CE-router, all the MPLS/VPN communities are lost and you get external OSPF/EIGRP routes. In that case, EBGP is a way better option. See also:

    http://blog.ioshints.info/2011/08/dmvpn-as-backup-for-mplsvpn.html

    ReplyDelete

You don't have to log in to post a comment, but please do provide your real name/URL. Anonymous comments might get deleted.

Ivan Pepelnjak, CCIE#1354, is the chief technology advisor for NIL Data Communications. He has been designing and implementing large-scale data communications networks as well as teaching and writing books about advanced technologies since 1990. See his full profile, contact him or follow @ioshints on Twitter.