Who the **** needs 16 uplinks? Welcome to 10GE world!
Will made an interesting comment to my Stackable Data Center Switches article: “Who the heck has 16 uplinks?” Most of us do in the brave new 10GE world.
A bit of a background
Most data centers have a hierarchical design, with top-of-rack (or leaf) switches connected via a set of uplinks to aggregation (or core or spine) switches, and the performance of your data center network depends heavily on the oversubscription ratio – the ratio between server-facing bandwidth and uplink bandwidth (assuming most server traffic traverses the uplinks).
Alternatives include full mesh design, monkey-see-monkey-do design, and novel approaches I can’t discuss yet.
Going from GE to 10GE
Most ToR switches we were using to connect Gigabit Ethernet server NICs to the network had 10GE uplinks, and the oversubscription ratios were reasonably low, ranging from 1:1.2 (various Nexus 2000 fabric extenders) to 1:2.4 (Juniper EX4200, HP 5830-48).
Some 10GE ToR switches have only 10GE ports (Brocade VDX 67xx, Cisco Nexus 5500, Juniper EX4500, HP 5920), and the Trident-based ones have a mixture of 10GE and 40GE ports (and you can use 40GE ports as 4x10GE ports with a breakout cable).
To maintain a reasonable oversubscription ratio, you have to use a quarter of the switch ports as uplinks (resulting in a 1:3 oversubscription) – sixteen 10GE ports in a 64-port 10GE switch or four 40GE ports in an equivalent 10/40GE switch. Regardless of the switch model you use, the number of fiber strands you need remains the same; 40GE link needs four fibers pairs.
Conclusion: if you want to have 1:3 oversubscription ratio, you need 16 fiber pairs to connect a 64-port 10GE ToR switch (or 48x10GE+4x40GE switch or a 16-port 40GE switch) to the network core.
Higher oversubscription ratios?
Do you really have to keep the oversubscription ratio low? Is 1:3 a good number? How about 1:7? As always, the answer is “it depends.” You have to know your traffic profile, workload characteristics, and plan for the foreseeable future.
Don’t forget that you can easily fit ~130 M1 Medium EC2 instances in a single physical server with 512GB of RAM. Assuming the server has two 10GE uplinks and you use 1:3 oversubscription ratio, that’s 51 Mbps per instance (ignoring storage and vMotion traffic). Is that good enough? You tell me.
You’ll find numerous fabric designs guidelines in the Clos Fabrics Explained webinar. Port densities and fabric behavior of almost all data center switches available from nine major vendors are described in the Data Center Fabrics webinar.
Both webinars are available as part of the yearly subscription and you can always ask me for a second opinion or a design review.
In some cases, the clustered storage is actually inside the virtualization hosts themselves (see Gluster or HP StoreVirtual VSA).
The capex and opex savings that come with eliminating FC are compelling, but the use of IP storage does require lower oversubscrition ratios. Fortunately Ethernet ports and links are (comparatively) cheap, especially when the outrageous costs of "vendor-blessed" HBAs, FC optics, FC switches, and monolithic FC SANs are included in a TCO model.
To my knowledge, approximately zero public/hybrid cloud service providers are using FC storage. Some might still offer FC with their managed/dedicated hosting offerings, but everything new is cloudy and IP-based.
I know what you mean: Two fully equipped, plus SFPed, plus licensed, plus carepacked Brocade port fibrechannel switches for 80000,-€ is simply too much nowadays!
I'm glad I have read this prior to a large 5548 and UCS implementation. Thanks a lot!
Yes, another way is smaller single-purpose fabrics if that's what you prefer, but not necessarily practical for converged private cloud infrastructure where the goal is ultimately "any workload" on a single infrastructure, but most problems have multiple solutions, ours is just one.