Building CsC-enabled MPLS backbone

Just got this question from one of my Service Provider friends: “If I am building a new MPLS backbone from scratch, should I design it with Carriers Carrier in mind?” Of course you should ... after all, the CsC functionality has almost no impact on the MPLS backbone (apart from introducing an extra label in the label stack).

Carrier’s Carrier architecture looks complex and scary (your customers can build their own MPLS/VPN networks on top of your MPLS/VPN service), but needs just a single feature on top of baseline MPLS/VPN architecture: LDP between PE and CE routers.

In traditional MPLS/VPN architecture you run IGP (or EBGP) between PE routers and CE routers and send IP packets across PE-CE link. CsC adds the ability to run PE-CE LDP, exchange labels for MPLS/VPN routes and send labeled packets across PE-CE link.

The LDP label advertised by the PE router is stitched to the label remote PE-router advertised over MP-BGP, building an LSP from the CE-router to remote PE-router. Likewise, the MPLS/VPN label advertised by the PE-router through MPLS/VPN gets stitched to the LDP label advertised by the CE-router, building an LSP from remote PE-router to local CE-router.

If the customer continues to run pure IP, you’ll still have just two labels in your MPLS core (MP-BGP label on top of LDP label), but the LSP extends beyond PE-routers, which can be useful if the customer wants to implement BGP-free core. CsC architecture also allows your customers to implement layered QoS – they can control both the Traffic Class (former Experimental) bits in the MPLS label and DSCP bits in IP header.

All of the PE-CE functionality is totally invisible to your P-routers. However, if the customer decides to offer MPLS/VPN service on top of your MPLS/VPN service, you’ll see an extra label in your MPLS core – the only precaution you have to make in your network design is to use larger MTU size (add at least 20 bytes – 5 labels – to the maximum MTU you want to support).

And you probably know where to find more information about CsC architecture – three guys wrote a great sequel to a great MPLS/VPN book a while ago.

11 comments:

  1. I dont want to sound picky, but i wonder when will we stop using MPLS VPN to describe L3VPN? isn't a VPLS a VPN running over MPLS? same for PW

    L3VPN might have been there first, or at least it used to be/is the most popular one, but we really had to get it out of our system by now :)

    ReplyDelete
  2. Probably worth mentioning that BGP RFC 3107 could be used for CsC PE/CE label exchange as well to form hierarchical LSPs. Theoretically RSVP-TE with forwarding adjacencies could be also used, but I haven't seen that option implemented.

    ReplyDelete
  3. Ivan Pepelnjak30 May, 2011 19:43

    Although there's more than a grain of truth in your comment, L3VPN is even worse (I don't know who invented that stupidity). A lot of things can be L3VPN, including IPsec, GRE, DMVPN and all 6/4 tunneling mechanisms.

    The official name of the technology we're talking about is "BGP/MPLS IP VPN". I guess I'll stick with MPLS/VPN, maybe explaining every time that I actually talk about RFC 4364 :-E

    ReplyDelete
  4. Ivan Pepelnjak30 May, 2011 19:51

    You're right, EBGP with labels works as well. RSVP-TE would work only with loosely source routed tunnels, as the VRF IGP doesn't have visibility into core IGP.

    The funniest part of CsC is that it's actually a single command (either to enable LDP or send labels with BGP), but the configuration guides describing that single command are 50+ pages long (yes, there are two of them, one configuration guide describing "mpls ip", the other describing "neighbor send-label" :-P )

    ReplyDelete
  5. Absolutely agreed, in my early MPLS days the greatest confusion was that every CsC scenario description begun with complex terminology and buzzwords as opposed to simply explaining the concept tunnel hierarchies - one of the MPLS fundamentals :) No wonder many people still think it's black magic.

    ReplyDelete
  6. I think there were attempts to properly differentiate L3VPNs are "overlay" and "peer-to-peer", though the latter term might sound too "loaded" due to proliferation of P2P term in late 90s/early 00's. The key feature of MP-BGP based L3 VPNs was control-plane interworking, while overlay VPNs featured just basic data-plane overlaying.

    ReplyDelete
  7. What would need to be kept in mind when designing a new mpls backbone specificly for csc? When designing a new MPLS backbone, maximizing MTU is already priority one as many L2VPN customers consuming VPLS service for DCI mandate 9000 (and sometimes higher) byte MTUs.

    CSC also seems to carry a very different service definition in the mind of the largest telco's. While all definitions include running LDP (or at the very least swapping labels on customer interfaces in a VRF) the routing protocol and thus design of the service differ.

    Some (like Sprint) want to run LDP with an IGP such as IS-IS to do label exchange and LSR prefix reachability.
    Others (like ATT) will only configure a static solution thus preventing dual provider multihoming for a CSC customer.
    I have heard about one provider in EMEA that performs BGP peering with label exchange which most other providers consider too dangerous.
    So while all of them acomplish the same thing (getting labeled packets to PEs that are customer controlled) by functioning in different ways carry very different configurations and operational headaches. Ever do inter-provider VPN label troubleshooting?

    The other consideration is that even without widespread CSC support, many customers are building their own MPLS services over provider networks. There is the payment processing giant FDC I mentioned before running 2547overDMVPN, a voice processing company running MPLS over L2 VPN (VPLS), and financial services such as citi just operating overlay VPNs on top of L3 VPN's. Overlay VPN technology like DMVPN or GETVPN work well when you dont want to expose internal routing to your service providers.

    Supposedly there was an MPLS over IP encap standard being written as well to address the headache of CSC, by making the L3 cloud in the middle completely agnostic and unaware of routing other than the connected endpoints.

    ReplyDelete
  8. Its great that your Service Provider friend was able to ask you that question, then get a blog post about it followed by several really useful comments!

    K.

    PS. In regards to those books, I hear one of those authors is a grumpy and crotchety old man who doesnt like anything. This has not been my experience tho.

    PPS. Also in regards to those books, read them from cover to cover. That way you wont miss a key concept that has you baffled. The answer was right on page 34 in a highlighted "NOTE" section. GRR!

    ReplyDelete
  9. Ivan Pepelnjak04 June, 2011 07:41

    " one of those authors is a grumpy and crotchety old man who doesnt like anything. This has not been my experience tho."

    Because you were never trying to sell me sexier heptagonish wheels ;)

    ReplyDelete
  10. But heptagon is the new round! Also out solution doesnt currently support rounded surfaces, but you can change your infrastructure to support a lot of straight edges instead tho, right? :-P

    ReplyDelete
  11. CsC for IPv6 is also great with eBGP+label between the PE-CE (RFC3107).
    It can be used if a MPLS-VPN Backbone is used to interconnect the POPs of an IPv6 Internet Service Provider. The IPv6 ISP can share the full IPv6 Internet routing table with a BGP session which get through the MPLS-VPN backbone but thanks to CsC, the MPLS-VPN does not have to learn the IPv6 Internet routes. The MPLS-VPN only carries the IPv6 ISP Infrastructure routes but not the Internet route.

    ReplyDelete

You don't have to log in to post a comment, but please do provide your real name/URL. Anonymous comments might get deleted.

Ivan Pepelnjak, CCIE#1354, is the chief technology advisor for NIL Data Communications. He has been designing and implementing large-scale data communications networks as well as teaching and writing books about advanced technologies since 1990. See his full profile, contact him or follow @ioshints on Twitter.