When Cisco ACI was launched it promised to do everything you need (plus much more, and in multi-hypervisor environment). It was quickly obvious that you can’t do all that on ToR switches, and need control of the virtual switch (the real network edge) to get the job done.
- 2017-03-17: It was not clear that the API I was referring to is the 3rd party vSwitch data plane API.
- 2017-03-26: VMware announced what I described here a week later. So much for "they will never dare to do it" wishful thinking from $NetVendor engineers.
Looks like even Nexus 1000V is currently not working on vSphere 6.5, and based on some VMware training material I’ve seen third-party distributed switches might not be supported at all in vSphere 6.5U1. Whether that means “don’t call us” or “they won’t work because the API will be gone” wasn’t exactly clear (so we’ll have to wait till 6.5U1 comes out), but with vDS being pretty close to Nexus 1000V feature-wise (and NSX competing with ACI) I would expect VMware to kill the internal vDS data/control plane API used by 3rd-party virtual switches like they killed the dvFilter API.
That would degrade Cisco ACI used in vSphere environments into a smarter L2+L3 data center fabric. Is that worth the additional complexity you get with ACI? It depends… and we’ll discuss some aspects of that in the VMware NSX, Cisco ACI or standard-based EVPN workshop in mid-May.
Note to potential commenters: you cannot implement EPGs and all the other fun stuff on granularity finer than VLANs unless you control the virtual switch. The proof is left as an exercise for the reader.
Update 2017-03-21: As is often the case I stand corrected (thanks to g9ais and Nillo) - you can use PVLAN to pull all traffic out of the hypervisor to the ToR switch, and process it there. I still think there are things that can go wrong with that approach with just the right mix of flooded traffic, so I don't think it's functionally equivalent to having full control of the virtual switch. Can't figure out at the moment whether that's relevant or not - any comment would be highly welcome.