Routing protocols running on virtual appliances significantly increase the flexibility of virtual-to-physical network integration – you can easily move the whole application stack across subnets or data centers without changing the physical network configuration.
Major hypervisor vendors already support the concept: VMware NSX Edge Services Router can run OSPF, BGP or IS-IS, and BGP is coming to Hyper-V gateways. Like it or not, we’ll have to accept these solutions in the near future – here’s a quick survival guide.
Don’t use link-state routing protocols
Link-state routing protocols rely on shared topology database flooded between participating nodes (routers). The whole link state domain is a single trust zone – a single node going bonkers can bring down the whole domain.
Conclusion: don’t use link-state routing protocols between mission-critical physical network infrastructure and virtual appliances. BGP is the only safe choice.
EBGP or IBGP?
I would usually recommend running EBGP between your network and a third-party appliance, but IBGP might turn out to be simpler in this particular case:
- IBGP sessions are multihop by default. We have yet to see whether virtual appliances support multihop EBGP sessions, and you probably wouldn’t want to establish peering between ToR switches and virtual apliances (see below);
- IBGP sounds more complex (you need route reflectors), but it’s usually perfectly OK to advertise the default route to the virtual appliance … or you might decide to use DHCP-based default routing in which case you don’t’ have to send any information to the virtual appliance.
- IBGP allows you to use MED and local preference to influence route selection if necessary.
Peer with a cluster of route servers
BGP configuration of a virtual appliance or a physical network device shouldn’t have to change when the application stack fronted by the virtual appliance moves into a different subnet. The virtual appliances should therefore peer with route servers using fixed neighbor IP addresses.
Here’s an anycast design that ensures a virtual appliance always finds a path to a route server regardless of where it’s moved to:
- Assign the same IP address to loopback interfaces of multiple BGP route servers and advertise these addresses with varying IGP costs (or you might get interesting results when ECMP kicks in ;). Obviously you’d use two anycast IP addresses for redundancy.
- When a virtual appliance establishes a session with the closest BGP route server, it announces its prefixes with the BGP next hop set to the physical IP address of the appliance. Assuming you run IBGP between your physical nodes, all routers in your data center get optimal routing information.
The route servers obviously have to accept BGP sessions coming from a range of IP addresses – dynamic BGP neighbors are a perfect solution.
If the routers or layer-3 switches you use don’t support dynamic BGP neighbors, use Cisco’s Cloud Services Router as a route server. It’s a bit more expensive than Quagga but also a bit more versatile.
Don’t trust, verify
You wouldn’t want just any VM that happens to be connected directly to a physical VLAN to have BGP connectivity to your route servers, would you? Use MD5 authentication on dynamic BGP sessions.
Likewise, you probably don’t want to accept routes at face value from untrusted nodes. Filter BGP updates received from virtual appliances, and accept only prefixes from specific address range assigned to virtual appliances having specific subnet size (for example, /64 in IPv6 world or /32 to /29 in IPv4 world).