Can VMware NSX and Cisco ACI Interoperate over VXLAN?
I got a long list of VXLAN-related questions from one of my subscribers. It started with an easy one:
Does Cisco ACI use VXLAN inside the fabric or is something else used instead of VXLAN?
ACI uses VXLAN but not in a way that would be (AFAIK) interoperable with any non-Cisco product. While they do use some proprietary tagging bits, the real challenge is the control plane.
In APIC release 2.0 you can run EVPN (a standard control plane) from an ACI fabric to a Nexus 7000 or ASR 1000 router.
NSX does use VXLAN but has features beyond ACI such as load balancers and firewalls.
Firewalls - yes, including distributed ones. Load balancers - yes, Edge Services Router can also work as a load balancer. Distributed load balancers? Works best in PPT; it's an awfully hard task to solve.
I understand VXLAN is a standards based protocol and NSX and ACI are vendor terms but how do they inter-relate?
VXLAN is just a data-plane protocol (like IP or Ethernet). The problem is the control plane - how do you figure out where everyone is (think OSPF or BGP). I think the intersection of NSX and ACI control plane protocols is still zero.
I heard recently in a VMware session saying you could use NSX with ACI and I scratched my head and thought why you would do this, what would this provide over the added support burden?
Sure you can. You build an ACI fabric to provide IP transport and run NSX on top of it ;) A bit expensive if you ask me ;))
However, I know production deployments running VMware NSX on top of Nexus 9000 switches using EVPN control plane. Mitja Robas talked about his experiences with one of those deployments in the last session of Autumn 2016 Building Next-Generation Data Centers online course (and of course you’ll get access to the recording of his presentation if you register for the Spring 2017 session).
Just like HiGig2 or other internal switching encaps, it doesn't leave the ACI fabric. Whether it's VXLAN or VLAN (or untagged), the packets get decapsulated and re-encapulsated into that iVXLAN header as it bounces around inside ACI. That header is removed by the time it leaves the ACI fabric. Each bridge domain is its own iVXLAN segment. Cisco refers to this as normalization, so that different encaps can be used on the same network.
MP-BGP (e)VPN is used for the new multi-pod topologies, and as far as I know right now it's only ACI-specific.
Then there's IETF standard VXLAN encap, which ACI can also do. There's some integration with vShield and OVS (as well as AVS) with either IEEE VLAN tagged frames or IETF standard VXLAN tagged packets, but I don't think there's currently any specific integration going on with NSX. In terms of ACI and NSX, I believe ACI would be an IP underlay.
A different story is with VXLAN in NX-OS standalone mode. Here VXLAN on Cisco can interoperate with NSX directly composing a single overlay network.
I thought also about a use case of running ACI over NSX. It could an encryption which is not a part of ACI and is included as L2VPN encrypted service in NSX. ;)
From the ACI side, you can do the network layer with plain VMWare covering the server side of things. I think you can even do bare metal deployments and layer-2 integration with Cisco's AVS... not 100% certain yet. But I don't think you need the advanced features of NSX on top of this. And like NSX, the VXLAN/EVPN stuff happens in the background. You do some early configuration and provide some info, then it all runs in the background like magic.
I don't know if you can (or should) be trying to merge the VXLAN fabric of either vendor with anything else, even something that uses RFC-standard VXLAN. I think that would be opening a can of worms. I suspect either platform would not gracefully accept VXLAN move/add/change coming from any external platforms, but I could be wrong.
Again, I really can't see what technical driver would ever lead to spending the money for both ACI and NSX. Even if you had two separate existing environments and suddenly had a business driver force you down the path of merging them, I think you would want to select which vendor is going to handle what part then strip out any overlapping features (if possible) so that your recurring maintenance and license renewals wouldn't be obscene.
In terms of cost, ACI is just 5-10% more than the equivalent Nexus 9K EVPN fabric, and only 20-30% more than the equivalent VPC/STP design. That's a wash, easily justified by ACI's single point-of-management alone! Plenty of ACI customers buy it expecting nothing more than a really good L2/L3 fabric.
When you run NSX on ACI, ACI is purely an underlay. But it's a damn good one -- (a) discrete fabric-wide security, QoS, and health stats for NSX VXLAN, ESXi vMotion, ESXi storage, and VMware management traffic; (b) BGP/OSPF peering with the NSX DLR for fully distributed/anycast L3 ACI-to-NSX routing (no VPC/dynamic routing goofiness); (c) MAC/IP routing with sub-millisecond (!) fault recovery across hundreds of leaves; and so on...
In contrast to ACI's chump change, NSX doubles your per-VM cost (based on several real-world scenarios I've done with and without ELA's). As a result, many NSX customers are limiting their deployments to narrow high-security areas, such as PCI, HIPAA, or IP.