Anycast

Whenever I start digging into technical details, I learn something new. A few days ago I've stumbled across the term anycast, which is a very interesting way to solve scalability issues:

  1. Deploy geographically dispersed servers using the same IP address (obviously they would also need a unique IP address or you wouldn't be able to manage them);
  2. Advertise your service as residing on that IP address (for example, use the IP address in NS records for DNS zones you host)
  3. Advertise the IP address (or corresponding IP prefix) into the Internet from multiple locations.
Although you could (theoretically) use anycast technology for any application, it works best with simple request-response UDP applications (for example, DNS). In all other scenarios, the application session would fail if the nearest anycast server would change following a change in the network.

6 comments:

  1. You know, the myth that anycast and tcp don't mix well is just that, a myth. There are many many large anycast clouds serving thousands of tcp sessions.

    A couple of papers on anycast + tcp:

    http://www.nanog.org/mtg-0606/pdf/matt.levine.pdf
    http://www.netcat.ca/tag/tcp-anycast/

    It is true that if the node goes down that the tcp sessions would fail, however, that is the case if you're using anycast or not.

    ReplyDelete
  2. I want to agree and expand upon Patrick's comments - multi-tier Web applications typically don't carry a lot of state on the front-end in the first place, and, secondly, if you're experiencing so many fluctuations in your IGP that this would be problematic at the TCP level, you've larger problems which need solving first, anyways.

    The best way to do anycasting is to run a routing daemon on the host, so as to minimize the risk of blackholing traffic in the event of a host outage.

    ReplyDelete
  3. The problem with TCP-over-Anycast is that the change in global BGP state (nothing to do with the stability of the servers or IGP) could redirect the end-user's traffic to another Anycast server, resulting in disrupted TCP session. The presentation Patrick mentioned in his comment admits and addresses this concern in numerous ways:

    * Your upstream ISP has to provide effective BGP communities (#4)
    * Coasts and countries appear to provide enough padding for route instability (#6)
    * Nodes that are near by could possibly require state between each node if routes are unstable (#6)
    * There's also the "Gotchas" slide (#20)

    However, the actual statistical data they've provided (#19) indicate that the failure rate is so low that you can ignore it, meaning that although my reservations were valid, they were mostly academic :) Thanks for pointing that out.

    ReplyDelete
  4. In most cases, BGP things which happen outside one's own network won't have any effect at all on which TCP front-end server ends up servicing the requests. But as stated above, and as you note, this is firstly rare, and secondly, given modern applications architectures, won't matter if it happens, anyways.

    ReplyDelete
  5. One popular use of anycast is in IP Multicast to support the redundant rendezvous points (aka Anycast RP). See this white paper below:

    http://www.cisco.com/en/US/docs/ios/solutions_docs/ip_multicast/White_papers/anycast.pdf

    It's using MSDP to make Anycast RP possible.

    ReplyDelete
  6. You should use anycast whereever you ask the customer to hard code an IP address. This allows you to alter your topology at something faster than the rate of change of your slowest customer.

    Anycast is also commonly used for DNS and RADIUS. For an introduction on DNS Anycast see http://www.gdt.id.au/~gdt/presentations/2006-07-18-linuxsa-anycast/

    ReplyDelete

You don't have to log in to post a comment, but please do provide your real name/URL. Anonymous comments might get deleted.

Ivan Pepelnjak, CCIE#1354, is the chief technology advisor for NIL Data Communications. He has been designing and implementing large-scale data communications networks as well as teaching and writing books about advanced technologies since 1990. See his full profile, contact him or follow @ioshints on Twitter.