One of my readers left this question on the blog post resurfacing the idea of running BGP between servers and ToR switches:
When using BGP on a VM for mobility, what is the best way to establish a peer relationship with a new TOR switch after a live migration? The VM won't inherently know the peer address or the ASN.
As always, the correct answer is it depends.
Supporting live VM mobility
If you want to support live (hot) VM mobility across ToR switches, don’t run BGP with the ToR switch. Regardless of how well you fine-tune the setup, it will take at least a few seconds before the BGP session with the new ToR switch is established, making your service inaccessible in the meantime.
As I explained in another blog post (yes, it’s almost exactly three years old), you SHOULD run a BGP session with a route server somewhere in your network, preferably using IBGP to make things simpler.
To add redundancy to the design, peer the VM with two route servers.
Supporting physical servers
If your servers don’t move, but you still don’t want to deal with neighbor IP addresses or AS numbers, use one or more of these tricks:
- Configure the same loopback address on all ToR switches (I wouldn’t advertise it into the network, and you definitely don’t want it to become the ToR switch router ID);
- Establish BGP session between the physical servers and the loopback address using either IBGP (so everyone is in the same AS number) or using local-as on the ToR switch to present the same AS number to all servers.
Deploying Cumulus-enhanced Quagga on the servers is obviously a better option. For more details, watch the Leaf-and-Spine Fabric Designs webinar or the video in which Dinesh Dutt explains the enhancements they made to Quagga.
Supporting disaster recovery
Running BGP between the virtual machines and the network simplifies disaster recovery scenarios (and alleviates the need for crazy kludges like stretched VLANs). If this is your use case:
- Run a set of route servers in each data center to support live VM mobility within each data center;
- Use the same IP addresses and AS numbers across route servers in all data centers to enable to VM to connect to the route server in the local data center;
- Don’t advertise the shared IP addresses between data centers (you don’t want the VMs to connect to a route server in another data center due to a crazy routing glitch).
Need even more details?
I’m sure we’ll discuss them in the Building Next-Generation Data Center course. The September 2016 session is sold out, but you’ll get the recordings even if you register for the April 2017 one.
You can also use ExpertExpress to discuss the details of your design with me.
One ExpertExpress session is bundled with the Professional subscription so you don’t have to ask for the budget approval twice.