Follow-the-Sun Workload Mobility? Get Lost!
Based on what I wrote about the latency and bandwidth challenges of long-distance vMotion and why it rarely makes sense to use it in disaster avoidance scenarios, I was asked to write an article to tackle the idea that is an order of magnitude more ridiculous: using vMotion to migrate virtual machines around the world to bring them close to the users.
That article has disappeared a long time ago in the haze of mergers, acquisitions and SEO optimizations, so I’m reposting it here:
After I made a particularly snarky comment about an article that touted inter-data center (DC) virtual machine (VM) mobility as the ultimate tool to reach the 100% availability heavens (this is why that argument is totally invalid), someone asked me whether I don’t believe in workload mobility, disaster avoidance and follow-the-sun data centers. I am positive that some businesses have needs for all three above-mentioned functionalities, but I also know that live VM migration is not the right tool for the job.
Let’s focus on the most bizarre of the three ideas: using VM mobility to implement follow-the-sun datacenters. The underlying business requirements are sound and simple – moving the servers closer to the end-users reduces latency and long-distance bandwidth requirements. Reduced latency also improves response times and throughput (see also bandwidth-delay product). However, you cannot reach this goal by moving the virtual machines around the data centers; you simply can’t move a running virtual machine over long-enough distances.
The maximum round-trip latency supported by vSphere 4.0 is 5 msec. While the timing requirements have been relaxed a bit in vSphere 5.0, the maximum round-trip latency is still 10 msec – way too low to implement the follow-the-sun model (you need more than 100 msec to get from Central Europe to Ireland, let alone across the Atlantic or Pacific).
Even if you were able to move a running VM between continents, you’d still face a number of other challenges. Bridging (the traditional mechanism used to support long-distance VM mobility) over such distances is out of question; most layer-2 protocols (like ARP) would time out when faced with round-trip delays measured in hundreds of seconds. You might be able to support the VM mobility with LISP, but even that approach has a number of drawbacks until someone implements LISP within the hypervisors’ soft switches.
So, is it impossible to implement follow-the-sun datacenters? Of course not, Googles of the world have solved the problem more than a decade ago using DNS-based load balancing (or anycast) between data centers and local load balancing within the data center. You can also use AWS and create elastic resources based on geographic load distribution. Both approaches do have one thing in common: they rely on properly architected scale-out applications.
In short, if would be nice if some of the high-level consultants took some time to check product data sheets and laws of physics (like the speed of light) before selling totally impractical marketectures, but I don’t expect it to happen any time soon.
As described in the Data Center Interconnects and Designing Active-Active and Disaster Recovery Data Centers webinars, and the Scalability Rules book, the only solution that really works is a scale-out application architecture combined with load balancers.
"Thursday night I completely failed the core datacenter operations over to the recovery servers using a combination of Veeam Replication and VMware migrations that in the end, really didn’t need to happen."
Of course you can't tell at the time that it didn't need to happen, but VMotion was part of the DR plan and COULD have been necessary if conditions were slightly worse.
Also, I have a question. I don't know exactly how VMotion operates. Is it possible that the 10 ms RTT restriction might be relaxed in, say, three or four years?
As for RTT, it's actually the bandwidth-delay-product problem. You have to copy memory pages to the other vSphere host faster than the VM changes them and that's hard to do if you have low-bandwidth or high-latency link. Can it be done? Sure. Will they do it? I hope not ;)
While I do have customers, deploying 10G networks <100KM who plan to do live vMotion the truth is that for most customers pause/stop->sync->resume in new location is MORE than enough to meet the business need. And by accepting a 30, 60, even 300s window the complexity level goes WAY down and the distances supported go WAY up. And no unicorns are harmed ;-)
The actual results depend on the BW, delay and VM page change rate. You could probably get reasonable results with WAN acceleration (optimizing TCP and/or compressing vMotion data like F5 is doing) if the VM is not doing anything (in which case, why would you want to move it at all ;) ).