Load Balancing Across Multiple MPLS/VPN Providers

Arnold sent me an interesting challenge: he’s using two MPLS/VPN providers, with most sites being connected to both providers. He’d like to load balance the inter-site traffic across all PE-CE links – an easy task if you’re using RIP, OSPF or EIGRP as the PE-CE routing protocol, but he happens to be using BGP.

The network diagram Arnold sent me

The network diagram Arnold sent me

Cisco IOS implementation of BGP supports load balancing, both across multiple EBGP or IBGP paths (and even a combination of VPNv4 IBGP path and VRF EBGP paths), but it’s pretty picky – the BGP paths have to be almost identical, including the contents of the AS path list. BGP will not use alternate paths if they’re not passing through the same autonomous systems.

The default behavior usually makes sense. AS path length is a very coarse indicator of distance and often doesn’t relate well to expected performance.

You can use the (still undocumented) bgp bestpath as-path multipath-relax router configuration command to relax the path selection rules. After entering this command, the router compares the AS path lengths (they still have to be the same), not the actual content of the AS path attribute.

Caveats

  • Make sure the uplinks have equal bandwidth and the upstream networks have comparable performance characteristics. Equal-cost load balancing across two unequal links or networks results in twice the performance of the worst-performing component.
  • BGP routing process will consider only almost identical paths (same AS path length, local preference, origin code, MED, IGP metric).
  • You cannot load-balance between an IBGP and an EBGP path (you might get forwarding loops). In the above diagram you cannot load balance between the edge routers in the regional office; IBGP-based load balancing has to be configured on core switches.
  • You won’t see the effects of the bgp bestpath as-path multipath-relax command in the BGP table. BGP still selects one best path, but is willing to copy more than one path into the IP routing table (where you should see multiple entries for the same prefix).
  • A BGP route reflector will not advertise more than one best path to its clients (unless you use BGP Add Paths functionality available in IOS XE 3.7S and IOS 15.3(1)T).

14 comments:

  1. I'm facing similar problem but I'm still not sure if I want to load-balance across the different providers because it will inevitably led to antisymmetric routing. There can be significant differences in latency (I can see up to 100ms difference between continents), jitter,... Anyone has experience what can be impact of this asymmetry to VoIP, Video, SQL, or any other sensitive traffic?
    Replies
    1. Hi Michal - Let me know how you achieved this. I am looking for similar solution. Please share how you are able to achieve the load balancing across the different service provider.
    2. Hi Michal and Rajeev,

      can you please explain how to achieve load sharing. i am using the same kind of network. but we are using both the links as " Primary - secondary" . Please help us to use both the links efficiently to get better throughput
  2. regarding the last bullet, one comment if i may. In order to have a path diversity in the RR in the first place you would usually need something like "best external" on the regional office peering routers
    Replies
    1. "Best external" would speed up the convergence @ RR (because the alternate path would already be available), but even if you configure "additional paths" the RR would not advertise the backup path (advertised by one of the edge routers) to its clients, because the backup path would be significantly different from the best one (or the edge router would not prefer the iBGP path and you wouldn't need the best external).

      Hmm ... maybe I should do a short video explaining these concepts ;)
    2. Actually, i was referring solely to the AddPath feature and not to this or any particular scenario ;) just meant to note that AddPath and "best external" are usually tightly coupled - and they truly are!

      sorry if the context of my comment wasn't clear
    3. Humm, depend on how you have configured your MPLS cloud. If you use the same RD, you have to tweak the RR to let it advertise more than one path. However, if you are using uniq RD and melting the routes with RT import/export, the RR will advertise all routes and not only the best ones. I that case advertise best-external will work.

      Moreover, you may achieve load-blancing on the Regional Office providing a back to back link between the CE and using multipath eibgp in a VRF lite context on the VRRP/HSRP master CE.
    4. The RD trick works great in MPLS/VPN environments (and yes, best external would definitely be propagated to all PE routers in that scenario), but not in pure IP forwarding or IP-in-global-table ones.

      Also, for EIBGP multipath you need MPLS forwarding toward IBGP next hop and MPLS label attached to the IBGP route (that's the only way to prevent forwarding loops), so I don't think VRF lite context is enough; you'd probably need full-blown MPLS/VPN.
  3. assuming the requirement to use both paths is to increase the available bandwidth between the ROBO & HQ / DC locations this poses a challenge in the event of the loss of one of the paths as you would immediately loose ~ 50% of your capacity which could adversely impact up on users / applications (though QoS may reduce this for key functions).

    A potential alternate may be to use Traffic Engineering to move certain traffic patterns to the second provider such as backups, bulk transfers, email, internet access, etc.

    additionally depending on the mpls carriers service offering the use of scavenger class for this traffic in the event of a link failure as part of a QoS policy may provide a more robust if admittedly more complex solution
  4. Another way to solve the AS-path issue: run a dual DMVPN mesh (one over each carrier), run BGP over the tunnels (and if he desires, stop running BGP with the carrier, leaving nothing but the transit links in the carrier's VPN table). Then he also really gets to reap the benefits of using two carriers, since he'll stop relying on the carriers' ability to withdraw prefixes when they should (and he'll control his own fate wrt choosing which network is blackhole-free at any given time). :)

    (or if he's using a platform with multi-hop BFD, use that along with multi-hop BGP from spoke to hub, advertise prefixes to carriers with no-export. Either way, solve the AS-path problem and actually gain the resilience you thought you were buying with two carriers, in return for some possibly-annoying complexity)
    Replies
    1. http://blog.ioshints.info/2010/12/where-would-you-need-gre.html ;)

      Also read http://www.cs.washington.edu/education/courses/csep521/07wi/prj/matthew.pdf for "more" details.
    2. I've seen the first link - where do you think I got the idea but from this blog (although we started running things that way at my previous $dayjob circa 2008; I have the feeling it was an earlier post that I intellectually pirated)? :) Next I'll start suggesting he run IPSLA/EEM from the spokes over those same tunnels...

      I've not seen the second (although I have seen the first of his referenced papers), but I like very much a default position of assuming all L3VPN products are essentially Byzantine, if you care enough about your packet delivery.
  5. Sounds like a job best suited for Performance Routing (aka Optimized Edge Routing). Can also solve some of Michal's concerns around jitter and latency by keeping traffic balanced based on destination prefix and by monitoring for deterioration in latency conditions while also aiming to keep utilization roughly equivalent between the two links.
  6. How about using something like FatPipe?
Add comment
Sidebar