When All You Have Are Stretched VLANs...

Let’s agree for a millisecond that you can’t find any other way to migrate your workload into a public cloud than to move the existing VMs one-by-one without renumbering them. Doing a clumsy cloud migration like this will get you the headaches and the cloud bill you deserve, but that’s a different story. Today we’ll talk about being clumsy the right and the wrong way.

There are two ways of solving today’s challenge:

  • You could realize that your problem is IP address mobility and solve it by implementing routing on host IP addresses instead of routing on IP subnets (turning IP into CLNP, but that’s yet another story).
  • You could take the shiniest hammer in your toolbox (stretched VLANs implemented over L2VPN) and use it no matter what problem you’re trying to solve. It’s so much fun attacking Phillips screws with a sledgehammer.

Let’s start with the IP routing on host addresses camp:

Then you have the vendors with shiny hammers (or engineers believing in Flat Ethernet Everywhere mantra). The worst idea I’ve seen in this category is probably (now long gone) Nexus 1000v Intercloud - custom NICs in cloud VMs connecting to another cloud VM that acted as a virtual switch (linecard) for on-premises supervisor VM. Not surprisingly, it worked as reliably as trying to make a T-shirt out of spaghetti bolognese.

Slightly less convoluted is the stuff VMware is promoting these days: stretched VLANs using L2VPN in NSX-T, which turns out to be Ethernet-over-GRE-over-IPsec. Not only are you exposed to the usual flooding fun and occasional forwarding loop (with the slight difference that now you’re billed by the flooded packet by your friendly cloud vendor), you also have to deal with challenges like local egress which was solved in NSX-V but is still missing in NSX-T 2.x, and local ingress which is mostly unsolvable until you admit you have a routing not a bridging problem.

Not surprisingly, solving all these challenges with routing on IP addresses approach is a breeze… apart from the minor detail that no sane ISP would ever carry your host routes, but of course you can use LISP to solve that problem and get even better job security. After all, it’s just routing as usual with few extra tricks (more about that in another blog post).

So why doesn’t everyone use the same (seemingly sane) approach? As always, there are tradeoffs. All solutions based on IP routing give you unicast IP communication. No multicast for cluster discovery, no weird protocols, no IP address sharing. The networking world would be a better place if we could limit ourselves to what makes sense, but when you start promising your customers a seamless migration of whatever **** they’re running into a public cloud, you’re bound to deal with a few weird cases.

And then there’s another subtle challenge: when you move a running VM into another subnet, it has ARP cache populated with MAC addresses from the old subnet, so it won’t be able to reach anyone in the new subnet (including the first-hop router) unless you fake the MAC addresses somehow. Cue one Ethernet to rule them all jingle.

You could solve that challenge with cold VM migration (shut down, move, restart), or with an agent running on the VM (VMware Tools comes to mind) that would flush the ARP cache and the routing table (if so instructed) after the hot VM move, but that would require actual coding and testing work. Sometimes it’s much easier to write a marketing story to persuade everyone why using an inferior and risky solution is so much better.

Long story short: as always, understand what problem you’re trying to solve, find the best tool for the job, and use it correctly. Recycled and remarketed technologies (see RFC 1925 rule 11) might not be the best fit, regardless of how well they look in PowerPoint or work in demos.

Latest blog posts in High Availability in Private and Public Clouds series


  1. I can’t agree more but when you present tour idea to a bunch of people that follow only what vendors say you become like an idiot that try to reinvent the wheel.

    Long story short as you say: are you more experienced than vendor’s principal engineers?

  2. "when you present tour idea to a bunch of people that follow only what vendors say you become like an idiot that try to reinvent the wheel" << at which point you walk away and start looking for other opportunities. There's absolutely no way a reasoned argument will persuade religious zealots.

    "are you more experienced than vendor's principal engineers" << No. But...

    • Do I have a different perspective? Yes - they focus on building their products and inventing new technologies that will differentiate their products, I focus on stable networks.
    • Do I have different experience? Yes - they see crazy customer problems waiting to be "solved" (instead of incinerated with napalm), I see network meltdowns.
    • Do I have different priorities? Yes - they have to create the best possible tool based on 20 years of legacy code, I have to select the best tool for the job (regardless of which vendor gets the P/O).

    Also, assuming the vendor's principal engineers are focused on creating the best possible solution, there's a layer of marketing and sales between them and the customer, and those layers add a different spin - while I'm focused on long-term customer success and network stability, most of them have a product to sell and a deal to close.

    Obviously there are people who don't get the nuances like this, in which case it's better to spend your time and energy somewhere else.

  3. Something somewhat related:
    https: //anetworkartist.blogspot.com/2020/04/do-you-really-need-vxlan-in-modern-data.html
Add comment