VXLAN Limitations of Data Center Switches

One of my readers found a Culumus Networks article that explains why you can’t have more than a few hundred VXLAN-based VLAN segments on every port of 48-port Trident-2 data center switch. That article has unfortunately disappeared in the meantime, and even the Wayback Machine doesn’t have a copy.

Expect to see similar limitations in most other chipsets. There’s a huge gap between millions of segments enabled by 24-bit VXLAN Network Identifier and reality of switching silicon. Most switching hardware is also limited to 4K VLANs.

Based on that document he became concerned whether merchant silicon switches might be a good choice for his small data center:

I’ve had impression that in small data center environments (two sites, a few ToR, ~1000 VMs & max 20 ESX hosts) all Broadcom chipsets should be “good enough” for us even without support for single-pass VXLAN routing. Is it really so? Those limits could hurt even our small DC.

Realistically, what that document is saying is "if you're careless enough to have all VLANs configured on all ports, you won't be able to have more than 300 VLANs on every port of a 48-port 10GE switch". Honestly, I would be scared of having 300 VLANs on every server-facing switch port no matter what the chipset limitations might be... and why would you need 300 VLANs for 1000 VMs anyway?

If you need more than a few dozen segments, you should either use a hypervisor-based virtual networking solution (example: NSX), an orchestration system that synchronizes the needs of physical and virtual switches, or a single-image data center fabric that does that behind the scenes.

One of them is architecturally correct, the other one preferred by networking vendors telling you how you should keep supporting legacy infrastructure for the next millennium.

Numerous vendors have edge VLAN pruning solutions that try to pull information out of vCenter (VM Tracer, VM Tracker...); you’ll find them described in Data Center Fabric Architectures webinar. The same vendors usually integrate with other orchestration systems like OpenStack.

However, what you should do as the starting point is what I've explained in the Networking in Private and Public Clouds, Designing Cloud Infrastructure, and NSX, ACI or EVPN webinars. Figure out:

  • Who the data center infrastructure customers are (hint: application developers);
  • What they really need (as opposed to what they're asking for);
  • And finally, what problem you’re trying to solve.

You’ll probably find that those limitations aren’t as bad as they sound.

Notes

Recent posts in the same categories

5 comments:

  1. Unknown 30 March 2018 07:25
    True, every $vendor (you should Trademark this one Ivan) using a white box switch based on BRCM chips will hit that issue, which is not an issue by the way. But I remember other vendors having the same limitations with custom ASICs. We don't have infinite table sizes... Some chips are now programmable, you may want to look at what they can offer if you still want to have 300 VLANs on all server-facing ports. Debug, debug...
    Replies
    1. Ivan Pepelnjak 30 March 2018 08:00
      "True, every $vendor (you should Trademark this one Ivan) " << I should have trademarked "BUM frames" ;)

      I remember a vendor having 256 VLANs in total with custom ASICs. And no, it was not in the 90s ;))
  2. Sathish Kumar 02 April 2018 17:02
    "Expect to see similar limitations in most other chipsets. There’s a huge gap between millions of segments enabled by 24-bit VXLAN Network Identifier and reality of switching silicon. Most switching hardware is also limited to 4K VLANs"

    If tenant VLAN tag is not carried in VxLAN encapsulated packet it is possible to use the 24 bit address space across the DC network.AFAIk some implementations support excluding the VLAN tag.

    Orchestration and management of this network becomes tricky. Thoughts?
    Replies
    1. Ivan Pepelnjak 02 April 2018 21:09
      It's not the problem of 24-bit address space, it's the question of how many L2 segments an individual switch can handle. Obviously you can use different VNIs on different switches.

      Orchestration and management: http://blog.ipspace.net/2018/02/single-image-systems-or-automated.html
    2. JC 04 April 2018 03:06
      Speaking of VM tracker on nx-os, I know I mentioned its existence to you during a webinar, but after experimenting with it, the feature is pretty limited (can only be enabled globally on all turnk ports, can't configure a set of always trunked vlans, for your esx vmknics for example...) and seems to be almost orphaned within Bu/TAC (no one really knows about it)... gave up on using it...
Add comment
Home
Sidebar