Hmmm: Cloudflare's Automatic Return Routing
A while ago, I found the How Automatic Return Routing solves IP overlap article on Cloudflare’s blog. They evidently have a technology that addresses a pain point well worth solving (access to shared resources from clients using overlapping address ranges). I just hate how they’re selling it. Go read the article first; I’ll wait.
OK, here’s what bothers me: the “VRFs and NAT are bad” claims, while they use the same technology in disguise.
Let’s unravel what they’re doing:
- They use session information to select the forwarding table (VPN tunnel) used for return traffic. We called those forwarding tables VRFs.
- Regardless of how smart they handle session setup, they will inevitably hit the case of two clients with the same source IP address using the same source port1. Handling that requires remapping (at least) the source port. You can call that remapping whatever you want; I call it NAT.
So what they really do is well-orchestrated inter-VRF NAT at scale. I’m positive it’s a great solution that doesn’t deserve the negative hype they’re using to sell it.
Incidentally, they also proved that session-based forwarding works at scale. It would be nice to know how fast they can make it work and what CPU resources the solution consumes.
-
While that’s admittedly very rare, I’m positive Cloudflare’s engineers didn’t just shrug and said “We believe in the magic powers of statistics.” They’re experienced enough to have encountered the Birthday Paradox more than once. ↩︎
Sounds like F5 auto last hop where packets are returned based on mac address but in larger scale.
The way I read the Cloudflare blog post, they were looking for a solution to a specific problem. I'd assume they already had a per-customer Internet gateway with NAT that connected via tunnels to each site of the respective customer. Then a (potential) customer wanted to use the same IP addresses in more than one of their sites.
Common guidance when dealing with overlapping IP address spaces includes use of VRFs and NAT. VRFs can be used to keep the address spaces separate, and NAT can be used to employ a combined address space with unique addresses for each customer site for the shared resources (in this case, the Internet gateway). This would mean to introduce additional VRFs per site and additional NAT between the Internet gateway and the customer sites.
Instead, they extended their existing solution by utilizing the existing stateful NAT towards the Internet (which includes connection tracking) and the special circumstance that each customer site is connected with a tunnel to the Internet gateway. They replaced the routing lookup for return traffic with an additional entry in the connection information, i.e., the tunnel interface to send the return traffic. This uses existing information, i.e., no additional information like address space mappings or new VRF IDs are needed. They did not need to introduce per-site VRFs, and did not need additional per-site NAT.
This solution works here because the customer already separates their overlapping IP address space (they use remote sites for this), each site already has a direct tunnel to the Internet gateway, and the Internet gateway already uses stateful NAT with connection tracking. It also helps that Cloudflare implement their own packet forwarding infrastructure.
They could have added (more) well-orchestrated inter-VRF NAT functionality to their existing solution to make the destination IP based routing lookup for return traffic work with overlapping site prefixes (no theoretical reason why this should fail).
Instead, they removed the destination IP based routing lookup for return traffic. They removed per-site prefix information from the existing solution, because it is not necessary for the intended functionality. This enabled the solution to work with any site prefix, overlapping or not.
And while I still claim they do something equivalent to inter-VRF NAT (maybe replacing client NAT with client interface in the flow table), I completely agree with this ;)
And yes, we need more of this type of creative thinking (and less marketing hype and FUD).
While what they do could also be done using inter-VRF NAT, I would not call their solution "equivalent", because it is more specialized than inter-VRF NAT.