Video: Scale-Out NAT
Network Address Translation (NAT) is one of those stateful services that’s almost impossible to scale out, because you have to distribute the state of the service (NAT mappings) across all potential ingress and egress points.
Midokura implemented distributed stateful services architecture in their Midonet product, but faced severe scalability challenges, which they claim to have solved with more intelligent state distribution.
Nuage Networks took a different approach in the Virtualized Services Platform: they figured out that in most cases you don’t need a generic (hard to solve) solution, but one of two manageable subsets:
- Static 1:1 NAT between an external (service) IP address and an internal (VM) IP address. The mapping is static (no state to distribute) and can be moved to the hypervisor host on which the VM is running;
- Dynamic outbound PAT (port-and-address NAT) to give internal servers access to outside resources. The outside IP address is irrelevant, allowing each hypervisor host to keep independent state.
Outbound sessions traversing per-host PAT are lost after a live VM migration. Inbound sessions traversing static 1:1 NAT are not affected, as the NAT mapping and the outside IP address are moved together with the VM.
For more details, watch the Scale-Out NAT video from Scaling Overlay Virtual Networks webinar.
How do hypervisors advertise Public floating IP addresses? Even in Openstack the floating IP address is added as secondary public IP address to the interface. Do the hypervisors run some routing protocol?
At Midokura we take advantage of the fact that we can precisely determine the ingress and egress hosts of a flow. These hosts are bound to a manageable number, so we can efficiently replicate state across them. At really large scale we turn to partitioning: an arbitrary packet can ingress at N hosts, but packets of a particular flow are pinned to a partition of N. Some of the details are explained here: http://blog.midonet.org/introduction-to-mn-part-5-flow-state/