Does It Make Sense to Build New Clouds with Overlay Networks?

TL&DR Summary: It depends on your business model

With the explosion of overlay virtual networking solutions (with every single reasonably-serious vendor having at least one) one might get the feeling that it doesn't make sense to build greenfield IaaS cloud networks with VLANs. As usual, there's significant difference between theory and practice.

You should always consider the business requirements before launching on a technology crusade. IaaS networking solutions are no exception.

If you plan to sell your services to customers with complex application stacks, overlay virtual networks make perfect sense. These customers usually need multiple internal networks and an appliance between their internal networks and the outside world. If you decide to implement the Internet-facing appliance with a VM-based solution, and all subnets behind the appliance with overlay virtual networks, you're almost done.

Slide taken from Cloud Computing Networking webinar

Customers buying a single VM, and maybe access to central MySQL or SQL Server database, are a totally different story. Having a subnet and a VM-based appliance for each customer paying for a single VM makes absolutely no sense. We need something similar to PVLANs, and the only overlay virtual networking product with a reasonably simple PVLAN implementation is VMware NSX for Multiple Hypervisors.

If you want to use any other hypervisor/virtual networking platform, you have to get creative:

  • Use a single subnet (VLAN- or overlay-based) and protect individual customer VMs with VM NIC firewall (or iptables)

Slide taken from Cloud Computing Networking webinar
  • When using an overlay-based subnet for numerous single-VM customers, use a simple L2 or L3 gateway to connect the subnet to the outside world. Most overlay solutions include hardware or software gateways, and a 2-NIC Linux VM will easily route 1Gbps of traffic with a single vCPU.
  • Worst case, use small PVLANs. There's no need for large or stretched VLANs if every customer has a single VM, more so if you don't give the customers fixed IP addresses but force them to rely on DNS.

Need help?

Check out my virtualization webinars or get in touch if you need design review or technology recommendation.

The webinars to consider include:

Not sure which webinar to watch? Try yearly subscription.

But wait, there's more!

I will be talking about Software-Defined Data Centers and private cloud infrastructure @ Interop 2014 Las Vegas. See you there!


  1. Sorry for my ignorance, but if you have vmware NSX implemented, don´t you already have everything you need to use overlays ? what would be a reason to not leverage that ? or in other words, why would you use NSX if not to use overlays ?
    1. Check VMware NSX requirements, the prerequisite vSphere license, and its pricing. Not everyone will go for NSX ... but if you do have it, overlays are the way to go.
  2. Overlays provide the simplest path toward customer controlled networks. So, if only the network team will manage customer networks, then overlays are not necessarily required. However, if "self-service" is a core component of the offering, then overlays are required.
    1. You could implement self-service with VLANs. There are numerous cloud deployments (including many OpenStack ones) doing exactly that. Does it scale? Well, obviously not beyond 4K segments. Is it stable? Khmm ... Are people doing it? Sure.

      Keep in mind that things might work even though they aren't sexy or overhyped.
    2. Could vs. should...

      Networks in cloud models need to move toward L3 not L2. VLAN segmentation is a limitation and will come back to bite.

      The most mature cloud infrastructures are utilizing the physical network as message bus and allowing software endpoints to manage routing/firewalling/segmentation.
    3. The only requirement for "self service" is programability. I know I have built my fair share of multitenant solutions over the past 5 years based on automated provisioning.

      While I consider overlays to be the right answer, they are certainly not the only answer.

      Network programability and Overlays often go hand in hand, but they are not the same thing. They all form part of the larger Software Defined story.
    4. I agree with you - you CAN solve the problem using other technologies (and I know you did, and I talk about your solution in my webinars), but from the KISS perspective overlay networks are the right solution (they have the minimum number of moving parts).
Add comment