Common Services VRF with EVPN Control Plane
After discovering that some EVPN implementations support multiple transit VNI values in a single VRF, I had to check whether I could implement a common services L3VPN with EVPN.
TL&DR: It works (on Arista cEOS)1.
Here are the relevant parts of a netlab lab topology I used in my test (you can find the complete lab topology in netlab-examples GitHub repository):
module: [ ospf, bgp, vrf, vlan, vxlan, evpn ]bgp.as: 65000groups:_auto_create: Trueswitches:members: [ s1, s2, s3 ]hosts:device: linuxmembers: [ h1, h2, cs ]vrfs:spoke:links: [ h1-s1, h2-s2 ]evpn.transit_vni: Trueexport: spokeimport: hubhub:links: [ cs-s3 ]evpn.transit_vni: Trueexport: hubimport: [ hub, spoke ]
You’ll find a detailed explanation of various topology elements in the Building Layer-3-Only EVPN Lab blog post. That blog post also describes how to start the lab or use lab devices from other vendors.
Notes:
- I used two VRFs (hub and spoke).
- Hub VRF imports hub and spoke prefixes. Devices attached to the hub VRF (CS) can communicate with all other devices.
- Spoke VRFs import only the hub prefixes. Devices attached to a spoke VRF (H1, H2) can thus communicate with devices attached to the hub VRF but not with other spoke devices.
- The only exception to the above rule is the spoke devices attached to the same VRF instance. If you want to prevent all inter-spoke communication, attach every spoke device to a different VRF.
Here’s the relevant Arista EOS configuration. Note the mismatch between import evpn and export evpn values on S1, and multiple import evpn values on S3.
vrf instance spoke
rd 65000:1
!
!
interface Ethernet3
description s1 -> h1 [stub]
vrf spoke
ip address 172.16.0.1/24
!
interface Vxlan1
vxlan source-interface Loopback0
vxlan udp-port 4789
vxlan vrf spoke vni 200000
!
router bgp 65000
!
vrf spoke
rd 65000:1
route-target import evpn 65000:2
route-target export evpn 65000:1
redistribute connected
vrf instance hub
rd 65000:2
!
interface Ethernet3
description s3 -> cs [stub]
vrf hub
ip address 172.16.2.3/24
!
interface Vxlan1
vxlan source-interface Loopback0
vxlan udp-port 4789
vxlan vrf hub vni 200001
!
router bgp 65000
!
vrf hub
rd 65000:2
route-target import evpn 65000:1
route-target import evpn 65000:2
route-target export evpn 65000:2
redistribute connected
Test Results
Every PE device advertises a single IP prefix. IP prefixes carry the expected RT communities:
s3>show bgp evpn detail
BGP routing table information for VRF default
Router identifier 10.0.0.3, local AS number 65000
BGP routing table entry for ip-prefix 172.16.0.0/24, Route Distinguisher: 65000:1
Paths: 1 available
Local
10.0.0.1 from 10.0.0.1 (10.0.0.1)
Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
Extended Community: Route-Target-AS:65000:1 TunnelEncap:tunnelTypeVxlan EvpnRouterMac:00:1c:73:4e:42:aa
VNI: 200000
BGP routing table entry for ip-prefix 172.16.1.0/24, Route Distinguisher: 65000:1
Paths: 1 available
Local
10.0.0.2 from 10.0.0.2 (10.0.0.2)
Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, internal, best
Extended Community: Route-Target-AS:65000:1 TunnelEncap:tunnelTypeVxlan EvpnRouterMac:00:1c:73:a6:f6:a6
VNI: 200000
BGP routing table entry for ip-prefix 172.16.2.0/24, Route Distinguisher: 65000:2
Paths: 1 available
Local
- from - (0.0.0.0)
Origin IGP, metric -, localpref -, weight 0, tag 0, valid, local, best, redistributed (Connected)
Extended Community: Route-Target-AS:65000:2 TunnelEncap:tunnelTypeVxlan EvpnRouterMac:00:1c:73:01:aa:de
VNI: 200001
The hub VRF imports prefixes with RT 65000:1 and RT 65000:2. The hub routing table contains the expected three prefixes:
s3>show ip route vrf hub | begin Gateway
Gateway of last resort is not set
B I 172.16.0.0/24 [200/0]
via VTEP 10.0.0.1 VNI 200000 router-mac 00:1c:73:4e:42:aa local-interface Vxlan1
B I 172.16.1.0/24 [200/0]
via VTEP 10.0.0.2 VNI 200000 router-mac 00:1c:73:a6:f6:a6 local-interface Vxlan1
C 172.16.2.0/24
directly connected, Ethernet3
The spoke VRF imports only prefixes with RT 65000:2. As expected, the spoke VRF on S1 (or S2) contains only two prefixes:
s1>show ip route vrf spoke | begin Gateway
Gateway of last resort is not set
C 172.16.0.0/24
directly connected, Ethernet3
B I 172.16.2.0/24 [200/0]
via VTEP 10.0.0.3 VNI 200001 router-mac 00:1c:73:01:aa:de local-interface Vxlan1
Finally, I added validation rules to the lab topology and ran the validation tests:
-
However, you might be the only one in the world doing it and might hit unexpected bugs. I might also be overly pessimistic; if you’re using something similar in production, please leave a comment. ↩︎
Thank you for pointing out that this is still a thing. I learned about those hub/common/services networks during my early days as a network engineer running a MPLS network. Unfortunatly I've never seen it anywhere else, despite it being a really cool "trick". Thaught it a lot of other engineers who where learing about MPLS and BGP and it makes me happy seeing it's still alive with EVPN. Thank you.
I think we deployed it at one customer who wanted to push all inter-site traffic through a centralized firewall. The hub site advertised the default route and collected the routes from the spoke sites.
Hi Ivan,
Out of curiosity, for the design you implemented for the centralized firewall, was your VPN label allocation for the HUB VRF done per prefix or per CE allocation?
With per VRF VPN label allocation, the PE hosting the hub VRF would perform an IP lookup after popping the VPN label. Since it has all the routes to the other spokes, traffic between spoke sites would be possible without going through the firewall. Spokes could reach each other via the hub, as they are drawn by the default route.
I had a similar use case for a customer in Arista, where currently only the per VRF label allocation mode is supported. We had to be a bit more creative and ended up using two HUB VRFs: one for traffic coming from the spokes (HUB-IN) and one for traffic going to the spokes (HUB-OUT). The first HUB VRF would only export the default route learned from the firewall in MP-BGP, while the second VRF would only import the spokes' routes.
@Keanu: IIRC Cisco IOS used pre-prefix allocation as default in those days, so it was a no-brainer.
If one were to use the per-VRF label, then of course, one would have to use ingress and egress hub VRFs (the traditional hub-and-spoke architecture described in a few MPLS books)
Fun article and great to see support for this added in Netlab.
Full disclosure, I currently work as SE at Arista. Responding here because I was surprised for this not to be as well known as I had thought it would be.
We see importing/exporting of different services happening quite often in the field, and it is also fully supported (and tested) both from EOS perspective - and it's modelled in the AVD data models (and now in netlab - thanks :-D) to allow easy network-wide deployment methods.
Fun article and great to see support for this added in Netlab.
Full disclosure, I currently work as SE at Arista. Responding here because I was surprised for this not to be as well known as I had thought it would be.
We see importing/exporting of different services happening quite often in the field, and it is also supported (and tested) from an EOS perspective - as well is it modelled in the AVD data models (and now in netlab - thanks :-D) to allow easy network-wide deployment methods.
Thank you for the feedback! This has been available in netlab for quite a while (probably since we added EVPN support), but nobody ever said, "I wonder what would happen if..." ;)
I have designed several data center networks with common service VRFs (monitoring, backup, etc) but with regular MLAG L2 "fabrics" and MPLS routers on the stick. ACI do have a common tenant. I have been thinking of having the same topology with EVPN, so this post is very useful.
However, it seems that the transit VXLAN acts as something similar as a inner VRF label in an MPLS VPN network. Just wondering, maybe VPN hub&spoke topologies with EVPN SR-MPLS would be simpler and better supported by the vendors than EVPN VXLAN, guess that EVPN SR-MPLS is used by the WAN providers....?
> It seems that the transit VXLAN acts as something similar as a inner VRF label in an MPLS VPN network
Yes, it's effectively per-VRF inner label.
> maybe VPN hub&spoke topologies with EVPN SR-MPLS would be simpler and better supported by the vendors
You'd have to talk to vendor PLMs, my vague personal impression from the outside is that everyone is focused on VXLAN in the data center, and that MPLS gets implemented when a big enough customer pushes the vendor to do it (Cisco is an obvious exception because it pushes SRv6).
Thanks for another very instructive post.
I considered your approach a while ago trying to replace the above discussed common services VRF based on MPLS L3VPN with EVPN.
Challenge is when only a default route is advertised from the common services VRF. - In MPLS we used per-prefix label allocation; label action on PE for 0.0.0.0/0 happens without route-lookup so label gets popped and traffic sent to the firewall. (desired scenario) - In EVPN there is no label switching; route lookup for destination happens on hub and traffic is directly sent back to a spoke without traversing the firewall.
So without deploying another layer of complexity with PBR or "IN-VRF" and "OUT-VRF" to redirect traffic to the firewall, how would you solve this in your scenario?
I can't see another solution but IN-VRF and OUT-VRF :(