One of the publicly observable artifacts of the October 2021 Facebook outage was an intricate interaction between BGP routing and their DNS servers needed to support optimal anycast configuration. Not surprisingly, it was all networking engineers' fault according to some opinions1
There’s no need for anycast2/BGP advertisement for DNS servers. DNS is already highly available by design. Only network people never understand that, which leads to overengineering.
For example, in October 2021, 118 sites advertised the J name server, some of them locally (to adjacent ISPs), others globally. Furthermore, while there are over 1000 instances of root name servers worldwide, at least RIPE often implements a K name server instance as a cluster of servers advertising the same IP address to the adjacent routers. Local nodes are implemented with two servers (even smaller hosted nodes have a single server), while the larger global nodes always run on a server farm.
I guess there’s no doubt root name servers use anycast. It must be the incompetent networking engineers setting them up like that, right? There might be other reasons beyond the maximum size of a DNS reply packet – the Evaluating The Effects Of Anycast On DNS Root Nameservers article (HT: @hugoslabbert) lists three goals of root name server anycast:
- Increased resilience
- Higher (improved) performance
- Reliability (deploying root name servers closer to clients).
There’s another reason organizations like CloudFlare use anycast: DDoS prevention. If you advertise the same IP prefix from hundreds of locations, it’s hard to build a large-enough botnet to bring them all down.
Closer to the DNS clients, many organizations use anycast DNS servers as recursive resolvers. Google’s 220.127.116.11 and CloudFlare’s 18.104.22.168 are the obvious examples; I’m positive any decent-sized ISP does the same. @rot26de described a great reason for that:
In the past I anycasted a (small) ISPs Resolvers. Linux ALWAYS waits for a timeout of the first resolver before asking the second6. That can cause loads of false monitoring timeouts of services relying on reverse DNS.
OK, so we know root name servers use anycast, and many recursive resolvers use it to ensure better performance for suboptimal clients, but surely an authoritative DNS server for a small organization does not need to use anycast?
Of course not. In the past, I had one name server running within the organization, and a secondary one running somewhere else to increase the resilience of name resolution7.
To be realistic: those days are long gone, I’m too old for that **** and use hosted DNS service… and all hosted solutions I would consider use anycast. It’s amazing how influential those damn incompetent over-engineering networking people got in the meantime.
Finally, consider the amount of DNS traffic any member of the FAANG club must be getting (Facebook claims to have 2.9 billion active users). Highly distributed anycast is the only sane way to survive that onslaught.
Want to know more?
- Best Practices in DNS Service-Provision Architecture is a must-read classic8
- I covered anycast as a load balancing technique in the Data Center Infrastructure for Networking Engineers webinar;
- Nat Morris had a nice Anycast on a Shoestring presentation at RIPE699 (video)
- Anycast DNS is used in most large-scale environments to provide resilience, scalability, improved reliability, and other features like geographic load balancing (for more details, listen to the podcast I did with NS1 in 2015).
- Your network or DNS traffic is not remotely similar to Facebook’s (or root name servers).
- Whatever lessons and experience you might have gained running your environment for ages might not be relevant in dissimilar-enough environments.
- Sometimes people do stuff for good reasons. It might be worth figuring out what their reasoning is even if you disagree with them – that’s how we learn and become better engineers.
Details removed to protect the overconfidently naive ↩︎
Advertising the same IP address from multiple locations, or having a set of servers accepting queries and responding from the same IP address. ↩︎
See also the first link in the Want to Know More? section ↩︎
Containing the IP address of the root name server ↩︎
Cisco IOS had the same behavior the last time I looked ↩︎
Even if my web site was down, it was nice to have working MX records to forward the mail to an alternate SMTP server. ↩︎
Found in a LinkedIn post by Jeff Tantsura ↩︎