One of my readers found this Culumus Networks article that explains why you can’t have more than a few hundred VXLAN-based VLAN segments on every port of 48-port Trident-2 data center switch.
Expect to see similar limitations in most other chipsets. There’s a huge gap between millions of segments enabled by 24-bit VXLAN Network Identifier and reality of switching silicon. Most switching hardware is also limited to 4K VLANs.
Based on that document he became concerned whether merchant silicon switches might be a good choice for his small data center:
I’ve had impression that in small data center environments (two sites, a few ToR, ~1000 VMs & max 20 ESX hosts) all Broadcom chipsets should be “good enough” for us even without support for single-pass VXLAN routing. Is it really so? Those limits could hurt even our small DC.
Realistically, what that document is saying is "if you're careless enough to have all VLANs configured on all ports, you won't be able to have more than 300 VLANs on every port of a 48-port 10GE switch". Honestly, I would be scared of having 300 VLANs on every server-facing switch port no matter what the chipset limitations might be... and why would you need 300 VLANs for 1000 VMs anyway?
If you need more than a few dozen segments, you should either use a hypervisor-based virtual networking solution (example: NSX), an orchestration system that synchronizes the needs of physical and virtual switches, or a single-image data center fabric that does that behind the scenes.
One of them is architecturally correct, the other one preferred by networking vendors telling you how you should keep supporting legacy infrastructure for the next millennium.
Numerous vendors have edge VLAN pruning solutions that try to pull information out of vCenter (VM Tracer, VM Tracker...); you’ll find them described in Data Center Fabric Architectures webinar. The same vendors usually integrate with other orchestration systems like OpenStack.
- Who the data center infrastructure customers are (hint: application developers);
- What they really need (as opposed to what they're asking for);
- And finally, what problem you’re trying to solve.
You’ll probably find that those limitations aren’t as bad as they sound.