L2 DCI with MLAG over VPLS transport?

One of the answers I got to my “How would you use VPLS transport in L2 DCI” question was also “Can’t you just order two VPLS services, use them as P2P links and bundle the two links into a multi-chassis link aggregation group (MLAG)?” like this:

Unfortunately, the VPLS service is never totally transparent. While you might get STP running across VPLS (but probably only if you ask), I would be extremely surprised if the CE-switches could exchange LACP/PAgP packets; these packets would be usually intercepted by the first switch in the carrier’s network.

Still, you can configure static port channel on the CE-switches and get MLAG across VPLS to work ... assuming, of course, that you’re willing to trust the Service Provider to bring down the PE-CE interface if something breaks anywhere within the VPLS cloud (otherwise your static port channel will turn into a nice partial black hole).

You could also try to run your own end-to-end tests. This is where my guesswork starts; I would appreciate comments from experts more familiar with switching platforms.

UDLD is probably out of question (yet again, one would expect UDLD packets to be received by the first provider switch), but Ethernet OAM might fit the bill. My question would thus be: can we run end-to-end Ethernet OAM on an interface that is part of a static port channel? Has anyone got that working?

More information

Numerous L2 and L3 DCI scenarios are described in the Data Center Interconnect webinar (register here). If you’d like to know more about the benefits and drawbacks of various VPN solutions (including pseudowires, VPLS, MPLS/VPN, IPsec-based VPNs and DMVPN), buy the recording of the Choose the Optimal VPN Service webinar.

Both webinars are also available as part of the yearly subscription package.

14 comments:

  1. Dmitri Kalintsev24 February, 2011 22:51

    Hi Ivan,

    If the reference is to my suggestion to use two VPLS instances, then I didn't have in mind running LAG over them. :) I thought of rather using them as two L3 links and doing whatever you normally would (like run your own MPLS on top of it). :)

    I usually don't recommend solutions which may or may not work; hence my suggestion to use either PBB or more radically OM-5130 (which runs GFP over Ethernet/IP), as these solutions don't have dependence on service provider's capabilities to pass L2 control protocols.

    Speaking of which, I did come across a few cases when an SP doesn't support Ethernet OAM, and on top of than will block your OAM frames, too.

    ReplyDelete
  2. LACP is for p2p links so VPLS is the wrong service for that. Even with an EoMPLS pseudowire the only L2 protocols a 7600 can forward are dtp, vtp, stp, cdp.

    ReplyDelete
  3. BTW I tested on 7600 with ES+ cards and LACP does not work with EoMPLS pseudowire or VPLS.

    ReplyDelete
  4. "I thought of using 2 VPLS instances as two L3 links" - but that's no fun; we know how to handle that.

    BTW, having two VPLS services from the same SP instead of one might slightly reduce redundancy (you can't survive two worst-case access-layer failures). Having two VPLS services from two SPs is obviously a totally different story.

    Last but definitely not least, obviously it's time to look into OM-5130. PBB does not solve the problem, it just introduces another layer of addressing (which does solve some other problems, but not this one).

    ReplyDelete
  5. So you can't have two parallel EoMPLS PWs bundled into a MLAG? Ouch ...

    ReplyDelete
  6. What about EoL2TPv3oVPLS with P2P-VPLS links...? I'm not sure if that's transparent enough for LACP/PAgP, but maybe it would work. However, such a "solution" is nothing I would recommend... ;)

    ReplyDelete
  7. Dmitri Kalintsev25 February, 2011 10:57

    > you can't survive two worst-case access-layer failures

    Care to elaborate? Depending on how big and fibre-rich your provider is (and how deep your pockets are), you could potentially get your four access links going to four different POPs, with no common elements whatsoever. With smaller providers, of course, there's risk that you'll land up sharing at least some infrastructure.

    > PBB does not solve the problem

    I guess in my definition of "solve" it does not - it is vendor-dependent. Some vendors allow creation of MAC-in-MAC p2p (UNI to UNI) tunnels, which don't implement any L2 processing on incoming frames (just encapsulate them and ship them over, no matter what they were - control or data). But yes, it is not universal so doesn't count. :)

    > it's time to look into OM-5130

    If you did look at my presentation from CEW APAC '10 (I posted link in one of my responses some time ago), there was a mention of work being done at ADVA. Last year I was working with their guys on feasibility of including MPLS-based tunneling into an NTU-sized device, and there was some progress. This reminds me to get back in touch with them to see what's happening, as this discussion is a clear use case. ;)

    ReplyDelete
  8. LACP over EoMPLS does work for 6500. 12.2(33)SXI3, Sup720-3BXL, 6748 w/DFC3. It will forward UDLD, STP, CDP to the other end.

    One thing I noticed was setting EXP via service policy would not affect UDLD,STP,CDP packets. They would be set to EXP 7 where as the rest of the traffic was EXP 5 per the service policy.

    To get decent reaction to one side of the one of the links going down, you need to use UDLD. Routing protocol would be preferred.

    ReplyDelete
  9. LACP doesn't work, but you could probably force it on. I can try tomorrow in our lab. As you already said though, you would risk blackholing traffic and I haven't played with CFM and OAM enough to know whether that could help.

    ReplyDelete
  10. #1: Assuming your SP hasn't deployed redundancy gear on your site, one failure in VPLS-A access link and one failure in VPLS-B access link (in the other site) would bring down the whole DCI.

    #2: Interesting. Which vendor(s) would that be? You know I'm too Cisco-focused.

    ReplyDelete
  11. See the comment from Killian; he got UDLD across EoMPLS, which should be enough.

    ReplyDelete
  12. Dmitri Kalintsev27 February, 2011 12:04

    #1: That would be a double failure, and having these two services from two different SPs won't be much help, no?

    #2: I'm reasonably sure Alcatel's ESS/SR can do it; and if I'm not mistaken some of Nortel's Ethernet gear could do it, too. Pretty sure there would be other implementations out there - mainly where the equipment wasn't designed as a switch from the beginning (like for example 7600/6500/Nexus) and implements bridging as a "value-add" function.

    ReplyDelete
  13. Dmitri Kalintsev06 March, 2011 21:57

    Hmm, funny - just looked at the CCIE Quick Overview Kit you've linked to today, and it noticed that L2PT now supports LACP/PaGP ("l2protocol-tunnel point to point" command). So there you go - you'll need three ports on your edge switch for each VPLS connection:

    1) port to connect to a VPLS
    2) exit port for QinQ
    3) your "edge" port (that you'd normally connect to a VPLS).

    Port 1 is connect to an SP's demarc;
    Port 2 is connected to Port 3.

    Port 1 carries only one VLAN - one you use on Port 2 as QinQ VLAN ID
    Port 2 is configured as QinQ with L2PT p2p for PaGP/LACP

    Replicate for the second SP VPLS and at the other DC.

    Configure 2 x Port 3 as a member of LAG.

    Pray it all works as expected! :)

    ReplyDelete
  14. Ivan Pepelnjak07 March, 2011 08:00

    There are all sorts of things you can do with loopback cables, but I would like to avoid them. They are probably no better than VPLSoMPLSoVPLS (there might be a marginal benefit depending on what your HW platform is)

    ReplyDelete

You don't have to log in to post a comment, but please do provide your real name/URL. Anonymous comments might get deleted.

Ivan Pepelnjak, CCIE#1354, is the chief technology advisor for NIL Data Communications. He has been designing and implementing large-scale data communications networks as well as teaching and writing books about advanced technologies since 1990. See his full profile, contact him or follow @ioshints on Twitter.