One of the networking engineers using my ExpertExpress to validate their network design had an interesting problem: he was building a multi-tenant VLAN-based private cloud architecture with each tenant having multiple subnets, and wanted to route within the tenant network as close to the VMs as possible (in the ToR switch).

He was using Nexus 5600 as the ToR switch, and although there’s conflicting information on the number of VRFs supported by that switch (verified topology: 25 VRFs, verified maximum: 1000 VRFs, configuration guide: 64 VRFs), he thought 25 VRFs (tenant routing domains) might be enough.

Another limiting factor is the number of prefixes that you can install in the TCAM, which is shared by all VRFs. The total number of all routes in all VRFs cannot exceed the TCAM size (24000 IPv4 routes or 6000 IPv6 routes on Nexus 5600).

2014-12-19: Rewrote the following paragraph - it seems Nexus OS supports multiple VRFs in the same OSPF instance.

The real roadblock turned out to be the routing protocol support. Nexus 5600 supports up to four OSPF or OSPFv3 instances, which is obviously useless if you want to run an instance per VRF. You can use the same OSPF instance for multiple VRFs, but are still limited by the number of neighbors. With fast OSPF hello timers Nexus 5600 supports up to 16 neighbors (no other information is available in the Verified Scalability guide), which is quite small as you have to run whichever routing protocol you choose in VRF-Lite mode on every virtual uplink (VLAN subinterface) of every VRF. EIGRP is no better: it supports multiple address families (VRFs) per routing process, but cannot have more than 50 active EIGRP interfaces.

It seems BGP is yet again the right answer. You can route multiple address families (VRFs) in a single BGP process, and configure per-VRF neighbors to peer with spine switches. The maximum number of active BGP peers supported on a Nexus 5600 with Nexus OS 7.0 is 256, which gives you four peers per VRF (for four spine switches) with 64 VRFs.

Not surprisingly, BGP is the routing protocol used to propagate customer reachability information in Cisco’s DFA.

More information

Need to know the limitations of individual data center switches? Check out the Data Center Fabrics webinar. Building a leaf-and-spine fabric? You’ll find the design guidelines in the Leaf-and-Spine Fabric Architectures webinar. Looking for a design review? Go for ExpertExpress.