IPv6 in a Global Company – a Real-World Example

More than a year ago I wrote a response to a comment Pascal wrote on my Predicting the IPv6 BGP table size blog post. I recently rediscovered it and figured out that it’s (unfortunately) as relevant as it was almost 18 months ago.

Other people have realized we have this problem in the meantime, and are still being told to stop yammering because the problem is not real. Let’s see what happens in a few years.

Pascal neatly described the problem faced by large multinational organizations when he wrote "If a large company has PI addresses, many branches, and use multiple ISPs, then there could be many /48 (one per branch) advertised to the ISP peers, unless the branches connect via vpn to the main site." Let's dig deeper into the problem.

This blog post is based on a real-life design problem, but I tried to make it generic enough to cover numerous similar design challenges. Also note that the problem is not specific to PI address space. Some large organizations decided to become LIRs just to handle the IPv6 morass.

Requirement#1: No NAT. IPv6 evangelists advertise the world with no NAT, which is by itself a good thing. NAT results in increased CapEx (more expensive boxes that have to maintain state at high speed) and OpEx (troubleshooting NAT-related problems).

Conclusion: Let's do it right - we won't use NAT in the IPv6 part of our enterprise network, which means we need global IPv6 addresses for every single host in our network. Not a big deal, it's pretty easy to get large chunk of IPv6 PI or PA address space if you can document the use case (/48 per location x number of locations worldwide).

If the remote sites use a single ISP, you could go with PA address space for remote sites and make the /48 assigned by the ISP their unique IPv6 prefix… until you have to change ISP and renumber (a project that usually mirrors the Titanic voyage).

Requirement#2: Ubiquitous redundancy. Every location must have connectivity to two ISPs. Remember, we're discussing a global organization - paying more for dual-homed business-grade Internet connectivity is way cheaper than dealing with problems caused by residential-grade access or non-redundant setups.

Implementing this requirement in a NAT-less IPv6 world usually requires a globally routable provider independent IPv6 prefix for every single site. Alternatively, one could build IPv6-over-IPv6 (or IPsec or SSL VPN) tunnels to regional hubs and use ISP-provided IPv6 addresses as underlay endpoints. However, there's the third requirement.

Requirement#3: Local Internet exit. Local offices of a global organization commonly access local web sites. It makes no sense to make their life harder by shuttling the traffic to a regional hub (over sometimes quite expensive international links - there are countries in the world where you still pay extra for international connectivity) and then back to a web site that might be located across the street from the local office.

Another requirement in networks that combine MPLS/VPN and Internet WAN connectivity would be fallback to regional hub in case of local Internet uplink failure. Also note that regional Internet exit isn't much different than local Internet exit.

There are two designs I could come up with that satisfy all requirements:

  • A network in which every single site advertises its own provider-independent globally routable IPv6 prefix to all upstream ISPs, effectively exploding the IPv6 routing tables world-wide.
  • In networks with a single local Internet uplink we could use multiple IPv6 prefixes on every subnet: ULA+global or multiple global prefixes (enterprise and Internet) with smart source address selection rules. Two or more Internet uplinks result in a mission impossible scenario.

The moment we allow some NAT (preferably in form of NPT66), the design becomes more realistic. We can satisfy requirements #2 and #3 by:

  • Using a single PI block;
  • Allocating /48 prefixes out of that PI block to all remote sites;
  • Using ISP-allocated prefixes as the public NPT66 prefixes;

However, we’re firmly back in the NAT land, but at least we’re using a stateless variant, which tends to be a bit cheaper and easier to implement.

Am I missing something? We’re talking about real-life implementations, so you don’t have to mention LISP until reasonable large number of ISPs deploy it in production in the global Internet.

14 comments:

  1. Hi Ivan,

    thanks, very timely & relevant post and heavily needed input to the ongoing debate about deaggregation & strict filtering. That said I'd like to confirm that an increasing number of organizations consider NPTv6 as "the way to go" for small/medium sites with local Internet breakout requirements. So, yes, from what I can see real-life points into that direction.
    btw: what is that LISP thing you mention? ;-)

    best

    Enno
    Replies
    1. "btw: what is that LISP thing you mention? ;-)"

      You don't want to know ;))
  2. "Requirement#1: No NAT."
    ...
    "The moment we allow some NAT (preferably in form of NPT66), the design becomes more realistic."
    Hmm, it seems difficult to stick to the rules ;)
    And why ask for a PI if you implement some NAT66?
    Replies
    1. "why ask for a PI if you implement some NAT66?"

      To avoid renumbering in the future when you change the ISP. Alternatively you could use ULA. Plenty of bits have been spent on this topic on my blog, Ed Horley's blog and Greg Ferro's blog.
  3. Hi Ivan, I've blogged a response, but essentially, what about RFC6296?

    Regards,

    Guy
    Replies
    1. ... which is what I mention in the last part of the blog post (NPT66), right?
  4. It isn't obvious from the stylesheet but there are two links to http://www.howfantastic.net/2014/10/23/ipv6-in-a-global-company-a-response-nptv6/ in my post above.
  5. Enter ideas from the ietf homenet working group: Source specific routing. A source specific gateway will forward packets from it's local /48 out the default gateway, and
    packets from an internal ipv6 network whichever way is defined, so a given network would, in this case, each machine would have a local address delegated from it's provider for internet access, and one or more addresses internally for transiting it's local networks. (chosen by prefix match)

    http://tools.ietf.org/html/draft-boutier-homenet-source-specific-routing-00 (multiple source specific routing protocol implementations in babel, ospfv3, and ISIS.

    Among other things, this solves the BCP38 problem for ipv6 thoroughly, and is still aimiable to a PI implementation.

    https://tools.ietf.org/wg/homenet/
    Replies
    1. Homenet is what is says: home net. Some of its ideas might not be applicable to remote sites of a large enterprise.

      Also, last time I checked, some parts of the world ran out of IPv4 address space year(s) ago. It looks like homenet hasn't produced a single RFC so far, let alone production-grade implementation that I could use. There seems to be a time gap between theory and reality.
  6. At what point is this deemed irresponsible, or at least not best practice? I'm planning one PI prefix per site, less than 100 sites initially, but eventually it could be thousands of sites..
    Replies
    1. Do you want to inject each PI prefix into the global routing table? Why?
  7. see requirement #1 and requirement #2 above
  8. I raised a similar question on the ripe ipv6-wg mailing list last year, there was a good answer from Gert Doering that touched on the second of your design alternatives.
    https://www.ripe.net/ripe/mail/archives/ipv6-wg/2013-June/002159.html
  9. Hi Ivan,

    I believe many global companies don't require each site connected to the Internet directly.
    To protect the communication to/from the Internet by a centrally managed FWs and IPS devices there are only several (tens) data centers providing regional hubs (as you call them under the Requirement#3).
    Wouldn't there a design described here
    http://www.cisco.com/web/about/ac123/ac147/archived_issues/ipj_14-2/142_multihoming.html
    under the "BGP Multihoming" section be a compromise solution?

    What I'm not sure though: Would one PI block be enough for a global company?
    Or would it need one PI block from RIPE for European sites, another from ARIN for American sites, etc.?

    Thanks,
    Milan Kulik
Add comment
Sidebar