MPLS/VPN Carrier’s Carrier – Myth or Reality?

Andrew is struggling with MPLS/VPN providers and sent me the following question:

Is "carriers carrier" a real service? I'm having a bit of an issue at the moment with too many MPLS providers […] Carrier’s carrier would be an answer to many of them, but none of the carriers admit to being able to do this, so I was wondering if it's simply that I'm speaking to the wrong people, or whether they really don't...

Short answer: I have yet to see this particular unicorn roaming the meadows of reality.

Carrier’s Carrier (CsC) seemed like a perfect idea for wholesale (transit) provides in the early days of MPLS – I know at least one of the big ones was pushing very hard to get it done. Transit providers could use CsC to offer their MPLS/VPN services as a transit MPLS backbone for other (smaller) MPLS/VPN providers.

However, even though CsC it was conceptually simple (all you had to do was to configure LDP or EBGP with labels on PE-CE link) I’ve never seen it working, and I’m guessing that most of the transit providers prefer offering VPLS (or other EoX) services – they’re way less hassle for the same money.

On the other hand, Carrier’s Carrier is nothing more than the ability to build your own MPLS/VPN service on top of someone else’s infrastructure, and probably won't solve your inter-provider issues. If you’re experiencing routing problems with your MPLS/VPN providers, stop bothering – use directly-connected subnets as transport endpoints, build GRE or MGRE tunnels on top of the providers’ MPLS/VPN network, and run your own routing protocols on top of them.

16 comments:

  1. So why the hell do we have to learn this stuff for the CCIE SP if no one uses it in the real world?
    :-)

    Kieran
    Replies
    1. It's called "creating mindshare" ;) If we push enough people to learn it, maybe someone will start using it :D

      I had the same mixed feelings about IP Multicast in the CCIE R&S exam.
  2. If one big service provider have MPLS/IP ( AS 100, for example) high-level network based on DWDM links and several MPLS/IP Metropolitan Area Networks in different cities, regions ( AS 200 ), than CsC will be a great tool to connect them via MPLS L3VPN.
    Replies
    1. That's a perfect (internal) use case. Do you know anyone doing it?
    2. Yes, but i cannot tell who exactly due to security reasons. One of the mobile network operators in Russia uses this technology in that way.
    3. I think this ISP with CsC runs it as a result of acqusition. For some reason I believe I know the network in question.

      That means if given ISP would have an option to plan the network from the very start (taking acqusitions into account), they would be more likely going for another architecture.
  3. I know of at least one deployment across a state government. The state is the super carrier and the divisions of the state (police, universities, etc) are the customers. Many of these sub-orgs are large enough to run their own MPLS networks.
  4. I have a different perspective, the opposite one - The fact that all major vendors support this feature means it has traction. the days where you released a feature in order to get traction are (nearly) over. today, the major carriers dictate most of the features and vendors follow to get deals. you can ask any PLM at any vendor how many features are postpone or cancelled because of business drivers.
    Replies
    1. I've seen plenty of RFPs in my life and the resulting networks, and the list of features asked for and actually used were widely different ;) ... and from the vendor perspective you have to support whatever your customers decide to throw into a kitchen-sink RFP requirement list.
  5. I do not know about any real implementation.
    But I believe such implementation must exist in large countries like US and Russia, where even large MPLS service providers cannot cover whole country with private optics, and are forced to lease lambdas or use other SPs VPN services.
    Certainly, layer2 VPNs will be much simpler solution for this situation, but again, for large SPs it can be an awful lot of point-to-point L2 VPNs, while VPLS will have a lot of drawbacks, so L3 MPLS VPN+labeled BGP with Carrier SP would perform best.
  6. I believe it is a security issue for the provider, as there is no way to do MPLS RPF check.
  7. It's not that "most of the transit providers prefer offering VPLS (or other EoX) services" but rather that it is way easier to sell Ethernet services (which are really *implemented* as VPLS or EoX, but that is transparent to the customer).

    From a network operations view, L3VPNs in general are much easier for the SP NOC to troubleshoot with the customer since the CE and PE do exchange some NLRI which can be viewed by both the SP and the Customer NOCs. Most customers do not really have the ability to troubleshoot Ethernet services due to both lack of knowledge and tools (and the belief that Ethernet just magically works). This situation could be improved if both customers and service providers used CFM as a diagnostic tool, but I have yet to see this in the real world either.
  8. In our multi network merger environment, existing MPLS networks have been migrated from having their own dedicated backhaul to a Carrier's Carrier model. Easier than migrating all the customers.
  9. FWIW, a couple years on, CsC is still not a commonly available service from our favorite public ISPs.

    Having said that, I will add that it is being deployed in atleast one less-than-public network - with mostly successful results thus far.
  10. I am personally responsible for designing the customer side of a CSC network to interconnect 10 POPs for our network. Used correctly, it can be a very strong design.

    1. The POPs will have some hefty multicast flow requirements between one another, making VPLS a poor option for the core carrier.

    2. CSC is very easy from a conceptual perspective. Its actually easier for us to implement CSC than if the carrier provided us a L2VPN. If they did that, I would either have to create a new IGP process for inter-POP communications, or use area 0 in across the world with non-area-0 in each POP (something like that). Then, I would have to either use seamless MPLS or leak PE loopbacks across IGP boundaries ... note that seamless MPLS and CSC are almost identical designs, except the former is for an intra-AS design :)

    3. Maximum isolation of failure domains. I can add a new POP later, run eBGP+Label with the core carrier, and there is no significant change to the other N-1 POPs notwithstanding the BGP routes they'll learn from the newly-added POP. With VPLS, assuming I extended IGP across the globe, that changes a bit. Customer also has a stiff requirement for isolation since failures in a POP should minimally affect other POPs.

    4. Label filtering is a breeze. I only want to allocate BGP labels for my PE loopbacks, while aggregating the networks of my POP transit links into something coarse (like a /16) so I can still have full IP reachability to assist my operations staff with troubleshooting. I suppose this is also true for LDP, but I only need to manage it at a single exchange point, rather than a big broadcast segment with many peers.

    Moral of the story is that just because a service is not commonly offered does not mean it lacks value. Likewise, just because something is popular/hyped-up does not mean it has value. Thanks Ivan for the great conversation :)

    -Nick
  11. I worked as a network engineer for a MPLS SP that uses its parent provider's MPLS network in some regions.
Add comment
Sidebar