EVPN: All that Glitters Is Not Gold

Cumulus Linux 3.2 shipped with a rudimentary EVPN implementation and everyone got really excited, including smaller ASIC manufacturers that finally got a control plane for their hardware VTEP functionality.

However, while it’s nice to have EVPN support in Cumulus Linux, the claims of its benefits are sometimes greatly exaggerated.

For example, David Iles from Mellanox claims that “… with EVPN, we have an industry standardized control plane for VTEP orchestration using an extension of BGP” and “… EVPN will deliver many of the promises of TRILL, FabricPath, VCS, and other data center fabrics but in a scalable, non-proprietary, way

Let’s get a few things straight:

  • TRILL is as standard as EVPN (if not more). Mentioning TRILL in the same list as VCS and FabricPath is disingenuous.
  • As much as I love EVPN, TRILL has fewer options and thus higher chance of having at least two interoperable implementations.

Finally, EVPN can be implemented in so many ways that I started calling it “the SIP of networking”. For example, at the moment (in release 3.2.1) Cumulus Linux implements only Type-3 routes (similar to what Cisco Nexus 1000V was doing) and relies on dynamic MAC learning, while Cisco’s and Juniper’s implementations on ToR switches use BGP-based MAC+IP address propagation with Type-2 routes.

2017-02-23: Cumulus Linux already supports type-2 (MAC/IP) routes. That was fast ;) However, I got a hefty dose of “these are all the things that can go wrong” information a few days back, and it's even worse than what I thought. More details in a future blog post.

I’m also hearing rumors that symmetrical IRB (Cisco) and asymmetrical IRB (Juniper) implementations (you’ll find more details in the Mixed Layer-2 + Layer-3 Designs part of Leaf-and-Spine Architectures webinar) don’t work well together… or maybe that’s just FUD - I would love to hear from someone who got EVPN working between a Nexus 9000 and QFX 5100 or QFX 10K.


  1. This is exactly why I don't buy into the rosy, shiny world of where everyone thinks that the entire network will be orchestrated and automated to the final piece of config on a plethora of vendors and it will all do interop beautifully! Did I mention there would also be peace on earth? ;)

    It can be difficult enough to have different types of optics to talk to each other. Is Ethernet always Ethernet? Yes, mostly but there are things to consider there as well. How good is inter-op in MPLS? Works, mostly, probably.

    Don't get me wrong, I like EVPN but sometimes even running protocols within the same vendor is challenging. An RFC can always be interpreted in different ways. When we do interop we often have to settle on getting the common features working which is less than either device might be capable of.

    Another concern is that if everyone started building networks exactly the same with exactly the same components. What would happen the next time we see something like Intel SOC issue? The consequences would be disastrous. So while some ideas are nice in theory not everyone can use them and too much homogenity would not be good.

    Think you have some good blogging material above :)
    1. Hi Daniel..

      Now you are talking about the old rosy "vendor lock-in" ....

      I know many networks here in denmark that have gone with a vendor / technology to circumvent vendor lock-in and they all end up with a mess of hacks and instability.

      From my point of view vendor lock-in is great as long as you build networks and compute like "lego pods" ... Some pods running blue lego others pods run green lego.
      All glued together with a fairly well tested multi vendor protocol vanilla BGP.
    2. Yes, exactly. That's how we have built networks for years and it works.

      I guess my point is that it will never make sense to build a leaf & spine topology consisting of Cisco, Juniper, Arista and Brocade all in the same topology. They would be different parts of the network and interconnected in some way. Different pods as you say.
  2. The problem with widespread take up of EVPN I'd the requirement for supporting protocols, minimum of PIM and some IGP.
    1. Layer-2 VPNs implemented with EVPN using either VXLAN or MPLS transport do not need IGP or PIM. Some implementations do. Another data point for my EVPN ~== SIP claim ;))

      Oh, and you should talk with those vendors and ask them why they want to pile so many unnecessary protocols on top of one another ;))
    2. Ivan wrote: "talk with those vendors and ask them why they want to pile so many unnecessary protocols on top of one another"

      The answer is pretty simple. Tens of thousands of customers world-wide want thousands of different protocols, different flavors, different behaviour, and different knobs.

      I'll give you a basic example: MPLS. There are customers who insist on solving every problem with MPLS. And then there are other customers who insist on keeping MPLS out of their network at all costs. What is a vendor supposed to do ? If they want to make a living, the only solution is to implement all those protocols and knobs. Or else they will have very few customers. Can you blame them ?
    3. Henk, I understand that (and immensely hate that reality), but there's a difference between "having to support all these protocols because customers" and "you MUST deploy PIM and IGP to run my variant of EVPN".

      The right answer would be "you COULD run EVPN on top of IBGP on top of IGP and use PIM to build multicast trees but you COULD also run EVPN with EBGP and source-node replication if you want to keep your DC fabric simple"
  3. Having built every routing protocol on the earth :-) - no implementation is as complicated as EVPN, L2 RIB, new API's between bridges and BGP, all the permutations and and communication between BGP/RIB/LM/bridge components, rate-limiters and new SM's, since CP =! FW. Huge amount of work on HW, workarounds for BCM, DF election, to mention just a few.
    No surprise - you are getting "circumcised" implementations from vendors without solid BGP and system know-how. I highly respect person who did EVPN for Quagga, however it is too much and too short time to build a fully functional code base.
    1. I'm really curious how long will it take before people realize that getting everything more and more complex is NOT the way to go. Following this trend for a few more years we'll end up in situation where operating a simple LAN would require an expert in rocket science.

  4. Brocade VDX switches have very good implementation of EVPN, we have tried that in our labs. It even supports all EVPN route types - L2 extension (MAC and MAC and ARP) and L3 - Extension ( Type 5 routes) it is useful for many DCI use cases
  5. "I’m also hearing rumors that symmetrical IRB (Cisco) and asymmetrical IRB (Juniper) implementations... don’t work well together."

    Would you expect them to? I'd have thought the node running symmetric IRB should be able to understand the routes sent by the asymmetric node (since it's just regular EVPN forwarding between PEs), but I wouldn't expect the asymmetric node to understand the routes sent by the symmetric node (since the forwarding is "between VRFs" in a sense).

    Is there a sensible use case that involves both symmetric and asymmetric IRB in the same DC?
    1. Totally agree with you. Node supporting symmetric IRB _should_ be able to interoperate with node that uses only asymmetric IRB, the other way will not work.

      Use case: using Cisco and Juniper gear in the same network ;)
    2. If we assume that IETF draft is some approximation of the respective implementations, then "asymmetric" side potentially might accept the routes and this information would be sufficient. "Symmetric" side would miss information required for L3 forwarding (and they promises to drop the route).

      On the data plane traffic from "asymmetric" to "symmetric" should have no problem (assuming route is accepted). "Symmetric" could have forwarded L2 traffic (but choose to drop route on contol plane), but L3 forwarding will definitely fail route lookup.

      It might be possible to do "asymmetric" routing on spine switches with something else in pure L2 mode as leaf nodes. But this is not exactly "interoperability", it is simply one of the properties of "asymmetric" implementation.
  6. Bit late to the party here, but having deployed VXLAN BGP EVPN on Nexus 9K (multi tenant & self service can be a valid use case for BGP-SIP :-)

    And I've upstreamed some Ryu patches to get Ryu to play along as well.

    Talking about MAC-IP: these can be injected into the fabric, but in my experiments Cisco opted to ignore the IP adres for L2 EVPN and instead uses unicast flooding ("ingress replication") while they could have simply accepted and ARP-table imported the IP-MAC from the BGP update.
    There's bound to be an explanation (and yes I have read the relevant RFC's, ugh) but I'm struggling to understand the downside of using the IP from the update.
  7. Does anyone saw working EVPN+VXLAN implementation between Cisco (N9K) and Juniper (QFX10K and 5110)? I'm preparing for such a POC and constantly I'm assured by both vendors representatives that there is a little to zero chance it will be working altogether. There is a plethora of differences in EVPN implementation of both about which I'm aware but still didn't have a chance to saw WORKING solution between both of these vendors.
  8. Awesome comments Daniel and Thomas, just because a technology pile becomes "standard"common sense shouldn't disappear. When I think of the complexity in operating multi-vendor networks, not sure where the advantage is. The approach Thomas mentioned is the only feasible way to really achieve multi-vendor. I remember some years back, ok, many years back where a customer of mine tried to build a multi-vendor STP-based network. The efforts of training the staff in the little differences and potential down-time because of misunderstanding was outperforming the initial financial attractiveness.

    There are significant efforts to make protocols interoperable. How long did it take to get MPLS working between multiple vendors? Jeff and Ivan can probably write a book about it :-)

    EVPN is still in its early days. While some vendors ship products for 2 or 3 years, many others just started the journey. The original contributors to EVPN had different use-cases in mind and so different implementation have been chosen, all compliant to the IETF RFCs or drafts.
    For the record: Symmetric/Asymmetric IRB is one of the differences and there are reasons why one was chosen over the other. There are others variations in implementation that exists and all are compliant to the RFC/drafts.

    Disclaimer: I’m working for Cisco and I’m very involved with the EVPN solution
  9. what I've heard from Juniper SE was that they have successfully implemented symmetric irb feature and was able to provide a proven multi-vendor interoperability (between cisco) at interop 2016.

    I am in a verge to rationalize this statement soon with my environment. lmk if anyone have done testing already.
    1. Yes, they have symmetric IRB now (at least on QFX10K, ask your SE about the other platforms), and yes, it seems most things work (at least in the interoperability lab).

      Based on what I've heard from someone who tried to deploy pretty complex EVPN setup on QFX10K with an early version of the code, test everything you want to use before committing to a deployment.
Add comment