Category: MPLS VPN
Paul Unbehagen made an interesting claim when presenting Avaya network built for Sochi Olympics during a recent Tech Field Day event: “we didn’t need MPLS or BGP to implement L2- and L3VPN. It was all done with SPB and IS-IS.”
During my ExpertExpress engagements with engineers building multi-tenant cloud infrastructure I often get questions along the lines of “How do I integrate my public IaaS cloud with my MPLS/VPN WAN?” Here are a few ideas.
A good friend of mine sent me an interesting question:
When I configure mpls ip on an interface, will all packets on that interface be labeled, or just the MPLS/VPN packets received through VRFs? I always assumed that stuff in the global routing table just got forwarded as IP packets without any labels.
Well, that’s not how MPLS works (at least not in its default incarnation on Cisco IOS).
One of the Expert Express sessions focused on an MPLS/VPN-based WAN network using OSPF as the routing protocol. The customer wanted to add DMVPN-based backup links and planned to retain OSPF as the routing protocol. Not surprisingly, the initial design had all sorts of unexpectedly complex kludges (see the case study for more details).
Having a really smart engineer on the other end of the WebEx call, I had to ask a single question: “Why don’t you use BGP everywhere” and after a short pause got back the expected reply “wow ... now it all makes sense.”
Have you ever tried to implement a large-scale DMVPN or MPLS/VPN network using BGP as the routing protocol? If you tried to stitch more than ~1000 sites together you’re well aware of all the pain caused by a small range of private AS numbers defined in RFC 1930. We can kludge our way around the limitation by reusing the same AS number on multiple sites (and using allowas-in when we need full routing information on every site), but such a design clearly sucks.
The next small step in my MPLS/VPN update project: EIBGP load balancing – why is it useful, how it works, why can’t you use it without full-blown MPLS/VPN, and what the alternatives are.
Andrew is struggling with MPLS/VPN providers and sent me the following question:
Is "carriers carrier" a real service? I'm having a bit of an issue at the moment with too many MPLS providers […] Carrier’s carrier would be an answer to many of them, but none of the carriers admit to being able to do this, so I was wondering if it's simply that I'm speaking to the wrong people, or whether they really don't...
Short answer: I have yet to see this particular unicorn roaming the meadows of reality.
Loads of niche features got crammed into (MP)BGP and MPLS since I wrote my MPLS books, most of them trying to tweak BGP (a scalable and reasonably slow routing protocol dealing with behemoth tables) to behave more like an IGP would.
It looks like we’ll never see updated versions of the books, so I’ll try to cover the new features with short videos. The first one on the list: BGP Best External – a mechanism that speeds up MP-IBGP convergence in primary/backup PE-CE scenarios using EBGP.
Arnold sent me an interesting challenge: he’s using two MPLS/VPN providers, with most sites being connected to both providers. He’d like to load balance the inter-site traffic across all PE-CE links – an easy task if you’re using RIP, OSPF or EIGRP as the PE-CE routing protocol, but he happens to be using BGP.
Matthew Stone encountered another unintended consequence of full Internet routing in a VRF design: the TCAM on his 6500 was 80% utilized even though he has the new Sup modules with one million IPv4 routes.
A closer look revealed the first clue: L3 forwarding resources on a Cat6500 are shared between IPv4 routes and MPLS labels (don’t know about you, but I was not aware of that) and half the entries were consumed by MPLS labels:
One of my readers sent me a surprising question: “We run only LDP in our MPLS network and need to run RSVP for TE and then phase out LDP. How could we do it?”
My first reaction was “Why would you ever want to do that” and I got no reasonable answer (suggestions, anyone?) but let’s focus on “Could you do it?”
TL&DR summary: You could, but that doesn’t mean you should.
Erich has encountered a familiar MPLS/VPN design challenge:
We have Cisco's 2901s with the data license running MPLS/VPN on customer site (the classical PE is at the customer site). Should we use eBGP between CPE router and network edge router, some sort of iBGP route reflector design, or something completely different?
The “it depends” answer depends primarily on how much you can trust the routers installed at the customer site (CPE routers).
One of my readers sent me an interesting problem a few days ago: the BGP process running on a PE-router in his MPLS/VPN network preferred an iBGP route received from another PE-router to a locally sourced (but otherwise identical) route. When I looked at the detailed printout, I spotted something “interesting” – the pre-bestpath cost extended BGP community.
During the February Packet Party someone asked the evergreen question: “Is it safe to run Internet services in a VRF?” and my off-the-cuff answer was (as always) “Doing that will definitely consume more memory than having the Internet routes in the global routing table.” After a few moments Derick Winkworth looked into one of his routers and confirmed the difference is huge ... but then he has a very special setup, so I decided to do a somewhat controlled test.
Whenever I’m explaining MPLS/VPN technology, I recommend using the same route targets (RT) and route distinguishers (RD) in all VRFs belonging to the same simple VPN. The Single RD per VPN recommendation doesn’t work well for multi-homed sites, so one might wonder whether it would be better to use a different RD in every VRF. The RD-per-VRF design also works, but results in significantly increased memory usage on PE-routers.
2012-07-24: Updated the conclusions based on feedback from nosx