Combining BGP and IGP in an Enterprise Network

Syed Khalid Ali left the following question on an old blog post describing the use of IBGP and EBGP in an enterprise network:

From an enterprise customer perspective, should I run iBGP or iBGP+IGP (OSPF/ISIS/EIGRP) or IGP while doing mutual redistribution on the edge routers. I was hoping if you could share some thoughtful insight on when to select one over the another?

We covered tons of relevant details in the January 2022 Design Clinic, here’s the CliffNotes version. Keep in mind that the road to hell (and broken designs) is paved with great recipes and best practices, and that I’m presenting a black-and-white picture because I don’t feel like transcribing the discussion we had into an oversized blog post. People wrote books on this topic; I’m pretty sure you can search for Russ White and find a few of them.

Finally, there’s no good substitute for understanding how things work (which brings me to another webinar ;).

Let’s start with the basics:

  • Contrary to what many data center networking vendors are telling you, you might not need BGP in your data center. Contrary to what many SD-WAN vendors are telling you, you might need it anyway. As always, it depends (more below).
  • Two-way redistribution is bad. It can be done, and if you’re willing to live with the resulting complexity, and want to have fun troubleshooting sessions when things go wrong, you could get it to work. For everyone else, I have just one word of advice: DON’T.

Do You Need BGP or Not?

Here are a few good reasons why you might want to use BGP in your network1:

  • If you want to advertise your address space to the public Internet – if you need redundant Internet connectivity, that’s the only sane way to go.
  • If you plan to run services that need BGP, for example EVPN. In that case, please use the simplest possible design (IBGP with IGP – more details). You’re not Microsoft, and don’t have the scaling challenges that prompted them to use EBGP as an IGP.
  • If you have too many prefixes and you’re hitting IGP scaling limits. In this case you might have to implement an edge IGP (to collect prefixes) and a different core IGP (to support IBGP). Does this sound too complex? It is; OSPF areas and route summarization might be a better fit unless you’re dealing with a giant organically-grown mess. Also, CPUs today are a bit faster than they were the last time I was facing this particular design challenge.
  • If you insist on having thousands of routers connected to the same subnet (typically Carrier Ethernet or DMVPN). IGPs are delicate souls, and were not designed to survive such an abusive behavior.
  • If you want to implement routing policies that are more complex than “use LTE for backup” which can be solved with link cost. In this scenario, BGP is usually warranted in large multinational core networks with expensive core links. Use IBGP within regions (where you don’t care about routing policy) and EBGP between the regions – AS path is a nice mechanism to keep track of the regions the traffic would have to traverse.
  • If you’re using MPLS/VPN services, and want to retain your sanity, run EBGP with the service provider. I covered that scenario in the Choose the Optimal VPN Service webinar.
  • If you’re running a routing protocol with servers or VM instances in your data center, use EBGP to build a trust barrier between network and hosts.2

Have I missed something? Please write a comment!

A Word on Redistribution

There are at least two ways to avoid the need for two-way redistribution (keeping in mind I’m painting a coarse black-and-white picture).

Run IGP in parallel with BGP. Carry all relevant prefixes in both protocols, or fine-tune what information you’re carrying in BGP. For example, if you need BGP to support EVPN, you might not need BGP IPv4 address family at all.

Make BGP your core routing protocol. Make sure you’re running BGP on every hop between any two BGP speakers3, and advertise default route or a set of aggregate prefixes into IGP. If that’s hard to implement, extend BGP to the point where it’s feasible to advertise a default route into IGP.

For this approach to make sense, you should have at least some EBGP in your network. The proof is left as an exercise for the reader.

For example, you might be running EBGP with your MPLS/VPN provider, and another EBGP session with your ISP. Run IBGP on every hop between the Internet Edge routers and CE-routers4. Advertise RFC 1918 summary prefixes into IGP from the CE-routers and the default route from the Internet Edge routers.

Alternatively, advertise the default route from all BGP speakers into IGP; the BGP router closest to the user(s) will pick up the traffic and sort it out.

Finally, if you’re running BGP between servers (or NSX-T edge gateways) and data center switches, there’s usually no need to propagate these more-specific routes beyond the data center edge – advertise a summary prefix (or a few of them) into the WAN network and move on.

Revision History

2022-03-29
Added routing on servers use case based on feedback from Sander Steffann.
2022-03-31
Added redundant Internet connectivity use case

  1. “Because my vendor told me to do so” is not one of them. ↩︎

  2. IBM mainframes supported only OSPF a long while ago, resulting in designs where a single link failure would inadvertently turn the mainframe into the most expensive (and slowest) core router you’ve ever seen. ↩︎

  3. Of course you could also build a BGP-free MPLS core, but we agreed to keep things simple, right? ↩︎

  4. Or stretch a VLAN between them. I don’t believe I just wrote that ;) out. ↩︎

2 comments:

  1. Another situation where I would use BGP is when deploying anycast services (like anycasted DNS resolvers) in the network. You don't want to have those anycast nodes interfere with your IGP. BGP provides a nice separation of control/policy to keep mistakes from bringing down the network.

    Replies
    1. Thanks a million -- forgot about that one.

      While it's perfectly possible to implement anycast with OSPF (see https://blog.ipspace.net/2021/11/anycast-mpls.html), I always recommend running EBGP with servers. Added to the blog post.

  2. Can you please explain for someone not as experienced why it is bad to use EBGP as an IGP? We are running edge OSPF for our data center and WAN sites and tying it together with EBGP (instead of IGP + IBPG core or OSPF route summarization as you described). It is working fine for us but I am trying to understand where the problem lies with that design.

    Replies
    1. And that's why I started with "I’m presenting a black-and-white picture" ;)

      There's nothing wrong with what you're doing if it works for you, and you're happy with whatever level of complexity you had to deal with to get it done. There's also the question of configuration defaults: EBGP in the data center is trivial with Cumulus and gets pretty messy with any other gear, in particular when you hit vendor limitations that push you toward IBGP-between-loopbacks-over-EBGP.

      You'll find tons of details in https://www.ipspace.net/kb/tag/BGP-DC.html

    2. Hi, the url specified in the comments (https://www.ipspace.net/kb/tag/BGP-DC.html) does not work. Could you please fix it? Thanks

    3. The content has moved to https: //blog.ipspace.net/series/dcbgp.html, and unfortunately, I can't fix a URL in an old comment 🤷‍♂️
Add comment
Sidebar