CE-to-CE IBGP Session in a Multihomed Site
One of my readers sent me a question along these lines:
Do I have to have an IBGP session between Customer Edge (CE) routers in a multihomed site if they run EBGP with the upstream provider(s)?
Let’s start with a simple diagram and a refactoring of the question:
- A multihomed site has two WAN edge (CE) routers
- Each CE-router runs EBGP with the adjacent PE-router.
- Do we need an IBGP session between CE-A and CE-B?
Our multihomed site is small enough to have a single L2 switch, and both CE-routers act as a default gateway for the attached hosts1. Now imagine a scenario where:
- CE-A receives a routing update for destination X from its upstream PE-router, but CE-B receives no corresponding update from its EBGP peer.
- A host sends a packet for X toward CE-B.
It’s obvious that CE-B should have the information that it can reach X via CE-A, and there are two ways to achieve that:
- Exchange the information over an IBGP session between CE-A and CE-B
- Redistribute EBGP information into an IGP (for example, OSPF)
As you might be running an IGP within the site and redistribute IGP information into EBGP anyway, you’ll quickly land in a two-way redistribution morass if you choose option#2. Running IBGP between CE-routers is a much better approach, and gives you the ability to have site-wide consistent routing policy. For example, you could use BGP local preference to indicate which paths should be preferred2, causing the other CE-router to prefer IBGP paths over EBGP ones.
Finally a word of caution: establishing an IBGP session between CE-routers that do not support RFC 8212 could turn your site into a transit site. Not fun if you happen to be a steel manufacturer attracting Cloudflare traffic. Make sure you have deployed outbound AS-path filters dropping transit paths on all EBGP sessions.
More Details
- Consistent routing within a multihomed site is one of the Expert Express Case Studies (disguised as an “MPLS/VPN + IPsec” question)
- We discussed redundant BGP-based Internet access in September 2022 session of ipSpace.net Design Clinic. In the same session we also discussed secure multi-homed customer BGP configurations.
- We’ll do a deep dive into securing BGP routing (including a brief history of well-known global routing FUBARs) in the Internet Routing Security webinar.
-
Ignoring for the moment the intricacies of first-hop redundancy protocols and ICMP redirects. ↩︎
-
Obviously you could also use a route map to set tags or metrics in OSPF type-5 LSA when doing BGP-to-OSPF redistribution, and use that information to set administrative distance of OSPF routes (assuming your device can do something as abhorrent as that) if your primary design goal is to have infinite job security. ↩︎
"causing the other CE-router to prefer IBGP paths over IBGP ones" Must be a typo
It is. Thank you, fixed.
Ivan,
I hope this IBGP neighborship will also help in forming an SDWAN Tunnel from CE-B to remote sites using ISP-A and from CE-A to remote sites using ISP-B?
Thanks, Viki
In such a small deployment I assume that both ISPs offer transit, so that both CEs would get a default route from their upstream.
In this case I would not iBGP the CEs together but have HSRP running on the two CEs and track the uplink (interface and/of BGP session) to determine the active gateway.
It's simpler, less memory intensive the convergence will be faster and you don't have to care about the GRT.
And if you wanted to IBGP them anyway, I would put a route-map on it to only exchange the default route from the upstreams, so that both CEs have a 0/0 route with different distance. The only thing I don't understand is in which failure scenario traffic would end up on a CE without an active BGP uplink.
pvh, not a good idea. Sometimes your ISP has routing failures during which all physical links and BGP sessions remain up, but no routes get advertised / exchanged. The only correct option is to establish an IBGP overlay, even if using FHRP. As the "site" grows it may require route reflection.