Q&A: Ingress Traffic Flow in Multi-Data Center Deployments
One of my readers was watching the Building Active-Active Data Centers webinar and sent me this question:
I’m wondering if you have additional info on how to address the ingress traffic flow issue? The egress is well explained but the ingress issue wasn’t as well explained.
There’s a reason for that: there’s no good answer.
The Problem
Let’s briefly describe the problem before going into the details. Assume someone forced you to deploy stretched VLANs (so you have the same subnet in multiple locations) or you only got a single /24 from your ISP or RIR (because that’s all that’s left and since nobody uses IPv6 yet going there is not an option – hope you noticed the sarcasm).
In both cases you have to advertise the same /24 from multiple data centers and the clients are confused: which way should they go? Here’s the diagram I used in the webinar:
Ingress traffic flow when multiple sites advertise the same prefix
Potential Solutions
If your problem is lack of address space, you could use anycast: advertise the same prefix from multiple data centers, but terminate the TCP sessions on a different set of load balancers, all using the same outside IP address. Works surprisingly well across the global Internet (just ask CloudFlare or LinkedIn).
If however someone forced you to implement a stretched subnet design, or you did it on your own because you trust your $vendor and you know you can make it work, you’re in deep **** anyway. There’s no good solution, particularly if the traffic has to traverse any stateful service (please don’t get me started on stretched firewall clusters).
I spent a lot of time a while ago describing the intricacies of redundant Internet connectivity (free video) and wrote a case study documenting how you might solve the external routing with L2 DCI, so if you need more details you’ll find them there.
Even More Details
On a more real-life front, Ethan Banks did a great presentation describing real-life lessons learned while operating active/active data centers during the autumn 2016 session of Building Next-Generation Data Center course (you get access to the recording as soon as you register for the course).