Repost: On the Viability of EVPN
Jordi left an interesting comment to my EVPN/VXLAN or Bridged Data Center Fabrics blog post discussing the viability of using VXLAN and EVPN in times when the equipment lead times can exceed 12 months. Here it is:
Interesting article Ivan. Another major problem I see for EPVN, is the incompatibility between vendors, even though it is an open standard. With today’s crazy switch delivery times, we want a multi-vendor solution like BGP or LACP, but EVPN (due to vendors) isn’t ready for a multi-vendor production network fabric.
Another issue with EVPN is cost in terms of price and project (delivery times). To have a robust & stable solution you should start with Trident III as if you go with Tomahawk I you will need re-circulation (and I would suggest avoiding that) and with Trident II+ you lack 100Gbps and 25Gbps, which it is the standard today. There are not many Trident III switches today in the secondhand market and a new one would take 1 year to be delivered. Without EVPN, you could obtain a Tomahawk switch for half the price and even lower latency if that nanosec difference really matters to you.
Another problem, that vendors have been solving lately with ranges, was the kilometer long configuration in EVPN and uniqueness with the RD for each VNI per switch, making the list of config even longer than in head-end replication.
And all these without mentioning the bugs that each vendor has when using EVPN on their switches…. Making it an unstable feature to turn on today. In the other hand, EVPN Multi-homing, on paper, solve lots of issues
There’s so much to unpack here, and I’d love to hear your feedback on some of the issues Jordi raised as it looks like I’m still living in a forward-looking bubble.
Hardware issues have nothing to with EVPN and all to do with routing in and out of VXLAN tunnels (RIOT). If you’re doing pure bridging, you should1 be just fine using Tomahawk-based switches regardless of whether you use head-end replication or EVPN.
Configuration verbosity is a huge deal unless you automate it (yeah, I know, I’m getting tiresome), although it’s not that bad on Cumulus Linux or if you use VLAN bundles2 on Arista. Adding layer-3 to the mix obviously increases the configuration significantly.
Vendor bugs have been a huge nuisance in the early days of EVPN – I knew a customer that needed an extra year to roll out a data center hardware upgrade because they insisted on using VXLAN with EVPN multihoming instead of head-end replication with MLAG cluster – but that was years ago. Are the bugs still haunting you?
Vendors claim they solved the interoperability issues, and proudly point to interoperability test reports to prove that life in the EPVN land is beautiful and rosy.
The reality is a bit different. As Dinesh Dutt pointed out in the Multivendor Data Center EVPN part of EVPN Technical Deep Dive webinar, things usually work if you’re using IBGP EVPN on top of IGP (OSPF or IS-IS) – the design promoted by almost no vendor3.
You might get an interesting blob of complexity if you blindly follow vendor recommendations and build EBGP-only fabric or run EVPN IBGP between loopback interfaces advertised with underlay EBGP. At the very minimum, you will have to configure route targets as every EVPN implementation uses a different approach to deal with route targets of EVPN instances spanning multiple autonomous systems. Finally, building a multi-vendor IBGP-over-EBGP fabric with EVPN-based multihoming seems to be a decent bet if you’re looking for a long-time job security.
-
I guess the exact value of should depends on the switch vendor and the underlying hardware. I wouldn’t be surprised to find a vendor that insists on using VXLAN RIOT the moment you configure EVPN. ↩︎
-
Attaching the same RD/RT to numerous VNIs – an approach that’s a perfect fit for the common every VLAN everywhere data center design. Unfortunately, some vendors don’t support VLAN bundles. ↩︎
-
…because it’s simple and works reasonably well in multi-vendor setup? Nah, that’s just a conspiracy theory 😜 ↩︎
At my last job we managed to achieve EVPN interop between Cumulus and Arista (due to different functionality not lead time) but we had to turn off Cumulus automagic and make the config more explicit. And yes, we "solved" the problem of generating voluminous configs using Ansible but that still leaves you open to typos. I guess that's why vendors are pushing "digital twin" simulations or config verification tools. Ultimately I think we need to move beyond text config files and templating but that's probably 10-20 years away for most people.
In retrospect I wish we had just built an L2 network.
The described situation is something that Apstra was designed to address as one of its key use cases. The interop matrix is of course not any-to-any, but is hopefully comprehensive enough to be useful.
To the best of my knowledge, all "Yes" intersection scenarios are tested for each of supported versions of the respective vendors' NOS.
https://www.juniper.net/documentation/us/en/software/apstra4.1/apstra-user-guide/topics/ref/feature-matrix-411.html
Wonderful. Add another abstraction layer to abstract away the mess we created in the first place 🤣🤣
That's why everyone loves the networking industry and IETF 😜
I'd look at it as an alternative to the in-house automation tools. While the outcome may appear similar, you do get the benefit of someone else building, testing, and supporting it.
One could argue that this lets your ops folks to focus on more interesting things, such as helping your in-house app developers / SRE with proper network and security designs and implementations.
I've done both and Apstra is 10x better than Ansible as long as you use it properly (you have to give up a lot of ego and nerd knobs). I agree with Dmitri that you should choose one management plane and stick with it so there aren't multiple management layers.
You can't compare Apstra and Ansible -- that's like comparing a car with its left front wheel ;)
IIRC Apstra did use Ansible to push the configurations to the devices a long while ago. No idea what they're doing now.
Regarding iBGP over eBGP "saga" let me repost a snippet from a comment I made in "The EVPN/BGP Saga Continues".
"... The grander EVPN story starts with its background and continued evolution across multiple domains and use cases. What is being called out here is a very specific data center reference design doc — there’s a story behind that too, and it’s not marketing.
... a key goal of that doc is to expose what our fabric controller is driving under the covers. It starts with the basic use case of a simple IP fabric. Some folks don’t need overlays. OSPF/ISIS is not ideal for very large fabrics so EBGP was chosen to avoid deploying different solutions for different size IP fabrics in the same company (think large enterprise or SP with many fabrics of different sizes geographically dispersed). ... However operators are free to replace EBGP with OSPF or ISIS if they see fit (and understand the flooding inefficiencies in large dense topologies). >> I started my comment in the original blog with this statement, which you ignored in this blog. <<
Then we added the overlay use case on top of the IP fabric use case solution. Many of our larger customers don’t want ANY overlay/tenant state in P routers (control or data plane) [i.e. rules out use of hop-by-hop EVPN route distribution]. So instead of the controller pushing different solutions for different customer types and fabric sizes (this too is complexity) we chose to keep it consistent given the outcome is the same in every case. In fact >>some customers have hosts in overlays and hosts in the underlay simultaneously<<, since only a subset of their endpoints need to be segmented away from the larger set, and/or they are migrating to host-based overlays or cloud-native application models.
The controller hides the verbosity (explicit config), but when operator has to troubleshoot, detail is there under the covers. ... We provided a doc that exposes what we do under the controller. That’s all. It’s not marketing. ..."
I assume the “the design promoted by almost no vendor” does include Arista, correct? We’ve pretty much only been recommended that for an upcoming SR-TE deployment by Arista and they even have super simple lab guides for that exact use case.
Oh shoot, I meant “does not include Arista”. Sorry!
I'm positive Arista has great lab guides and super-helpful SEs, but their official EVPN-with-VXLAN configuration documentation focuses on the crazy stuff -- the only example of IGP+IBGP I could find in "sample configurations" (https://www.arista.com/en/um-eos/eos-sample-configurations) is EVPN-with-MLPS stuff.
Not that it would be hard to figure it out, but one should always start with the easy stuff and work from there.