A bit of a background
Most data centers have a hierarchical design, with top-of-rack (or leaf) switches connected via a set of uplinks to aggregation (or core or spine) switches, and the performance of your data center network depends heavily on the oversubscription ratio – the ratio between server-facing bandwidth and uplink bandwidth (assuming most server traffic traverses the uplinks).
Going from GE to 10GE
Most ToR switches we were using to connect Gigabit Ethernet server NICs to the network had 10GE uplinks, and the oversubscription ratios were reasonably low, ranging from 1:1.2 (various Nexus 2000 fabric extenders) to 1:2.4 (Juniper EX4200, HP 5830-48).
Some 10GE ToR switches have only 10GE ports (Brocade VDX 67xx, Cisco Nexus 5500, Juniper EX4500, HP 5920), and the Trident-based ones have a mixture of 10GE and 40GE ports (and you can use 40GE ports as 4x10GE ports with a breakout cable).
To maintain a reasonable oversubscription ratio, you have to use a quarter of the switch ports as uplinks (resulting in a 1:3 oversubscription) – sixteen 10GE ports in a 64-port 10GE switch or four 40GE ports in an equivalent 10/40GE switch. Regardless of the switch model you use, the number of fiber strands you need remains the same; 40GE link needs four fibers pairs.
Conclusion: if you want to have 1:3 oversubscription ratio, you need 16 fiber pairs to connect a 64-port 10GE ToR switch (or 48x10GE+4x40GE switch or a 16-port 40GE switch) to the network core.
Higher oversubscription ratios?
Do you really have to keep the oversubscription ratio low? Is 1:3 a good number? How about 1:7? As always, the answer is “it depends.” You have to know your traffic profile, workload characteristics, and plan for the foreseeable future.
Don’t forget that you can easily fit ~130 M1 Medium EC2 instances in a single physical server with 512GB of RAM. Assuming the server has two 10GE uplinks and you use 1:3 oversubscription ratio, that’s 51 Mbps per instance (ignoring storage and vMotion traffic). Is that good enough? You tell me.
You’ll find numerous fabric designs guidelines in the Clos Fabrics Explained webinar. Port densities and fabric behavior of almost all data center switches available from nine major vendors are described in the Data Center Fabrics webinar.
Both webinars are available as part of the yearly subscription and you can always ask me for a second opinion or a design review.