Building Network Automation Solutions
6 week online course starting in September 2017

Do you really need to see all 512K Internet routes?

Last week the global routing table (as seen from some perspectives) supposedly exceeded 512K routes, and weird things started to happen to some people that are using old platforms that by default support 512K IPv4 routes in the switching hardware.

I’m still wondering whether the BGP table size was the root cause of the observed outages. Cisco’s documentation (at least this document) is pretty sloppy when it comes to the fact that usually 1K = 1024, not 1000 – I’d expect the hard limit to be @ 524.288 routes … but then maybe Cisco’s hardware works with decimal arithmetic.

In any case, networks relying on boxes that support 512K IPv4 routes in their switching hardware might face interesting problems in a very near future… and the workaround is: reduce the maximum number of IPv6 routes. Isn’t that just the perfect solution for the days when most of the world already ran out of public IPv4 addresses? On the other hand, the default IPv6 table size on Cat6500 is 256K entries, and the current size of global IPv6 routing table is below 20K (HT: Renesys blog).

The same hard limits usually don’t apply to boxes that perform packet forwarding in software, for example ISR-series routers and probably also ASR1K – but don’t forget that you’ll need more memory as BGP table, RIB and FIB continue to grow.

Regardless of the platform you’re using in your Internet edge, you should ask another question: do I really need the full Internet routing table? Most edge networks don’t, and I described several techniques you can use to reduce the BGP table size.

Furthermore, you can use BGP Selective Route Download (unfortunately available only on IOS XE) or a similar Junos functionality to reduce the IP RIB and FIB sizes (see also my article on virtual aggregation).

Need more help?

Some pretty large organizations got fantastic results after one or two ExpertExpress sessions. Why wouldn’t you join them?

6 comments:

  1. Quote from http://www.cisco.com/c/en/us/td/docs/ios/ipswitch/command/reference/isw_book/isw_i1.html#wp1087660
    The defaults are as follows:
    •For XL-mode systems:
    –IPv4 unicast and MPLS—512,000 routes (<-- This answers your question)
    –IPv6 unicast and IPv4 multicast—256,000 routes
    •For non-XL mode systems:
    –IPv4 unicast and MPLS—192,000 routes
    –IPv6 unicast and IPv4 multicast—32,000 routes

    To the number of IPv4 you should add OSPF/ISIS/static routes,VRF routes (even a "global" route takes an entry in the FIB) and MPLS labels.
    If you don't use IPv6 you should disable it completely and allow the IPv4 space to increase to 1M.
    You should plan an upgrade for this gear, but you can use a filter like "deny ARIN/8s ge /20" to squeeze one more month of you gear.

    ReplyDelete
    Replies
    1. Thanks for the comment, but we still don't know whether it's 512K translated into 512,000 by the documentation writer, or whether the TCAM actually holds 512.000 entries (which is perfectly possible, but would make little sense from the technology perspective).

      Delete
    2. The limitation comes from the 4GB of memory they install with the DFC cards. They need some memory for multicast and some space for service data and other control structures. They decided to use a 1000 multiplier so they guarantee that they have some memory left for those structures.
      I think that (I'll be using IEC numbers http://www.iec.ch/si/binary.htm):
      -IPv4 strucute uses 4KiB (4096 bytes)
      -IPv6 structure uses 8KiB (8192 bytes)

      512000*4KiB=2000 MiB (2097152000 bytes)
      256000*8KiB=2000 MiB (2097152000 bytes)

      4 "GB memory" = 4GiB = 4294967296 bytes
      Service data avaliable: 4 GiB - 2*(2000 MiB) = 96 MiB (100663296 bytes)

      Delete
    3. I think that you should check "show platform hardware capacity forwarding" (or "... pfc" on 6500) output for the right number (I don't have any 6500/7600s for 4 years now). Some 6500 outputs show (available after googling):
      72 bits (IPv4, MPLS, EoM) 524288 432158 82%
      You see the 512*1024.
      I still think that people having those problems were having more routes because of OSPF, VRF routes and/or MPLS labels. They consume the same space, thus leaving less space available for the big BGP table.

      Delete
    4. Now this makes perfect sense. Thank you!

      Delete
  2. 512K tcam work-arounds for all of the Cisco platforms:
    http://blogs.cisco.com/sp/global-internet-routing-table-reaches-512k-milestone/

    ReplyDelete

You don't have to log in to post a comment, but please do provide your real name/URL. Anonymous comments might get deleted.