Multi-Vendor EVPN Fabrics

Daniel left an interesting comment on my Studying EVPN to Prepare for a Job Interview blog post:

I also never build a VXLAN fabric with two vendors. So, is it possible now to build one large fabric consisting of multiple vendors?

TL&DR: Yes, but just because you could doesn’t mean that you should.

TL;DR generated by ChatGPT GPT-4 ?
EVPN allows for multi-vendor fabric, but it’s not without complications. To minimize issues, use a single vendor per layer, IBGP for EVPN address family, IGP for underlay routing, and avoid complex features like EVPN multihoming. Stick to supported implementations and avoid experimental configurations for a smoother experience.

EVPN is a complex technology full of optional nerd knobs, and early implementations quickly turned it into SIP of Networking1. Fortunately (for the rest of us), EVPN vendors eventually came to their senses and started participating in EVPN interoperability testing, usually done during the MPLS & SDN World Congress (2022 test report).

However, just because boxes from different vendors work together doesn’t mean we’re in a carefree plug-and-play world2. If you’re forced to build a multi-vendor EVPN fabric it makes sense to stay as far away as possible from the vendor shenanigans (like IBGP EVPN over EBGP underlay) and simplify your design:

  • Use a single vendor for a layer (leaf or spine). You might claim that’s cheating as all the hard work is done on the leafs, and the spines serve only as route reflectors (or not even that). It is possible to have mixed-vendor leafs, but it will make your life more interesting.
  • Use IBGP for EVPN address family. Most EVPN implementations use BGP AS number as part of the EVPN route target and while it’s possible to make multi-AS EVPN work it’s a needless hassle.
  • Use an IGP for the underlay routing. You’re not a hyperscaler3 and don’t need EBGP to scale your underlay routing. It doesn’t matter whether you use OSPF or IS-IS (most vendors have decent implementations of both), but you might want to stick with OSPF to be extra safe4.
  • Stay away from complex stuff like EVPN multihoming. It’s safer to use traditional MLAG with anycast VTEP.
  • Multicast BUM replication is not supported by all vendors. Stay with EVPN-driven ingress replication.
  • Use symmetric integrated routing and bridging. It’s supported by most vendors these days, although you might encounter a few anycast gateway quirks.

In the ideal world I wouldn’t have to write what comes next, but some people love to live dangerously and push the envelope, so here we go.

Don’t even think about:

  • Complex nerd knobs unless you need eternal job security.
  • Using stuff that is not explicitly supported. Don’t try to be overly smart. Your coworkers, your users, and vendor TAC engineers will appreciate that.
  • VXLAN-to-VXLAN bridging. Multiple vendors can do it, but you probably don’t want to be the one to figure out how to make them work together5.
  • Multivendor MLAG clusters. MLAG is more than just a few hacks in the EVPN route types; not all of its mandatory components are standardized.

For more details, watch the excellent Multivendor Data Center EVPN presentation by Dinesh Dutt.

  1. If you ever had to use DTMF tone generator because your VoIP gateway and the public VoIP exchange to which it was connected couldn’t agree on how to do DTMF signaling you know what I’m talking about. ↩︎

  2. We might have reached the plug-and-pray phase though. ↩︎

  3. And if you are, I hope you’re reading my blog for the snark, not to get fabric design advice. ↩︎

  4. Hint: EIGRP is a wrong answer. So is RIP, but for other reasons. ↩︎

  5. Once I asked a TAC engineer whether it’s OK to run ~20 EIGRP processes on a single router and he replied: “sure, just keep in mind you’d be the only one in the world doing it.” FWIW, that design was in production for over a decade before we managed to replace it with MPLS/VPN. ↩︎

Recent posts in the same categories

2 comments:

  1. Chinar Trivedi 20 April 2023 02:02

    Great Blog as usual, Ivan. I agree your OSPF Area0 P2P recommendation as Underlay, however what's your take on OSPF Unnumbered for building underlay? I am not talking of course in Multi-vendor OSPF-Unnumbered. Just in general? and how about BGP-unnumbered underlays with RFC-5549 ? just for building underlays?

    Replies
    1. Ivan Pepelnjak 20 April 2023 07:45

      OSPF unnumbered is perfectly fine if it's implemented correctly on all boxes. I got it working on Arista (required a tweak), IOS, IOS-XE, NXOS, Junos, Cumulus... Details in https://blog.ipspace.net/2022/01/ospf-unnumbered.html

      Unnumbered BGP is a totally different beast. It makes sense only with EBGP when the BGP configuration supports neighbor interface command (mostly FRR-based implementations), but that quickly results in all sorts of weirdness unless the vendor is willing to run EVPN AF over those same EBGP sessions. Run away from crazy stuff like EBGP-over-EBGP or IBGP-over-EBGP, or grab a bag of popcorn and read https://blog.ipspace.net/2020/02/the-evpnbgp-saga-continues.html and all the links (or https://blog.ipspace.net/series/dcbgp.html)

  2. Craig Weinhold 05 May 2023 04:34

    "Use a single vendor for a layer" is cheating and needs more nuance. E.g., "multicast routing will likely fail between leaves of different vendors" or "isolating leaves of different vendors by VRF or L3 can help avoid L2 interop problems."

    Replies
    1. Ivan Pepelnjak 05 May 2023 04:53

      I guess we're almost in sync.

      "use a single vendor for a layer" means "don't use leafs (or spines) from two vendors". No nuance necessary, just don't do it.

      Also, "Multicast routing will like fail" means you were not following the "Stay with EVPN-driven ingress replication" ;)

Add comment
Home
Sidebar