A networking engineer was trying to persuade me of importance of hardware VXLAN VTEPs. We quickly agreed physical-to-virtual gateways are the primary use case, and he tried to illustrate his point by saying “Imagine you have 1000 servers in your data center and you manage to virtualize 80% of them. How will you connect them to the other 200?” to which I replied, “That doesn’t make any sense.” Here’s why.
How many hypervisor hosts will you need?
Modern servers have ridiculous amounts of RAM and CPU cores as I explained in the Designing Private Cloud Infrastructure webinar. Servers with 512 GB of RAM and 16 cores are quite common and becoming relatively inexpensive.
Assuming an average virtualized server needs 8 GB of RAM (usually they need less than that) you can pack over 60 virtualized servers into a single hypervisor hosts. The 800 virtualized servers thus need less than 15 physical servers (for example, four Nutanix appliances), or 30 10GE ports – less than half a ToR switch.
Back to the physical world
The remaining 200 physical servers need 400 ports, most commonly a mixture of everything from Fast Ethernet to 1GE and (rarely) 10GE. Mixing that hodgepodge of legacy gear with high-end hypervisor hosts and linerate 10GE switches makes no sense.
What should you do?
I’ve seen companies doing network refreshes without virtualizing and replacing the physical servers. They had to buy almost-obsolete gear to get 10/100/1000 ports required by existing servers, and thus closed the doors for 10GE deployment (because they won’t get new CapEx budget for then next 5 years).
Don’t do that. When you’re building a new data center network or refreshing an old one, start with its customers – the servers: buy new high-end servers with plenty of RAM and CPU cores, virtualize as much as you can, and don’t mix the old and the new world.
This does require synchronizing your activities with the server and virtualization teams, which might be a scary and revolutionary thought in some organizations; we’ll simply have to get used to talking with other people.
Use one or two switches as L2/L3 gateways, and don’t even think about connecting the old servers to the new infrastructure. Make it abundantly clear that the old gear will not get any upgrades (the server team should play along) and that the only way forward is through server virtualization… and let the legacy gear slowly fade into obsolescence.