Is it safe to run Internet in a VRF?

During the February Packet Party someone asked the evergreen question: “Is it safe to run Internet services in a VRF?” and my off-the-cuff answer was (as always) “Doing that will definitely consume more memory than having the Internet routes in the global routing table.” After a few moments Derick Winkworth looked into one of his routers and confirmed the difference is huge ... but then he has a very special setup, so I decided to do a somewhat controlled test.

Results summary

Running Internet services in a single VRF versus the global IP routing table increases the overall memory consumption by around 8%, while providing clean separation between internal ISP network (in the global IP routing table) and public Internet (in the Internet VRF).

Configuring Internet services with multiple VRFs having different route distinguishers (for example, using Overlapping VPN or Common Services designs) dramatically increases the memory consumption due to the MPLS/VPN VPNv4 BGP route replication and multiple copies of full Internet routing table in per-VRF FIBs. Every additional VRF increases the memory consumption by ~80%.

Test setup

The difference between having global BGP and IP routing table, and having VRF-based BGP table (and VPNv4 prefixes) should be way more noticeable with large BGP table size, so I decided to announce a few hundred thousand BGP prefixes to a router running Cisco IOS release 15.1(3)S2.

I used the trick Jeremy Gaddis described a while ago, cursed bgp_simple because it would send a different number of BGP prefixes every time I ran it, and finally decided to settle for approximate results.

I executed show memory free before the BGP session with bgp_simple was established, show ip bgp summary to see how many BGP prefixes the router received from bgp_simple, and show memory alloc total to see how much free memory was left and which processes were the largest memory consumers.

Internet in the global routing table

I used the following simple configuration to establish the baseline case. The router had a global EBGP session with a Linux machine running bgp_simple.

interface FastEthernet0/0
 ip address 10.17.0.1 255.255.255.0
!
router bgp 65000
 bgp log-neighbor-changes
 neighbor 10.17.0.2 remote-as 65100

Prior to the test, the router had almost 360M of free memory:

Free memory on PE-A prior to the Internet in global routing table test

PE-A#sh mem free
               Head   Total(b)    Used(b)    Free(b)  Lowest(b) Largest(b)
Processor  64FB4E00  402960204   41332768  361627436  361627436  352628556
      I/O   E000000   33554432    3134628   30419804   30419804   30417180
Transient  7D000000   16777216       5156   16772060   16772032   16772028

bgp_simple sent the router 186,683 BGP prefixes (from a file containing 200,000 unique BGP prefixes). They consumed a bit more than 45M:

BGP memory consumption during the Internet in global routing table test

PE-A#sh ip bgp sum
BGP router identifier 172.16.0.1, local AS number 65000
BGP table version is 1, main routing table version 1
186883 network entries using 27658684 bytes of memory
186883 path entries using 11960512 bytes of memory
34942/0 BGP path/bestpath attribute entries using 4472576 bytes of memory
31632 BGP AS-PATH entries using 1555798 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 45647570 total bytes of memory
BGP activity 186883/0 prefixes, 186883/0 paths, scan interval 60 secs

Neighbor    V     AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.17.0.2   4  65100  186889       4        1    0    0 00:00:57   186883

BGP uses several memory structures to store the BGP table:

  • Network entries table (IP prefixes)
  • Path entries table (BGP paths)
  • Path attribute entries (Attributes associated with BGP paths)
  • AS-Path entries (unique AS-paths)

The memory consumption per BGP prefix obviously depends on the path attributes associated with the prefix (example: BGP communities), AS-path lengths and number of unique AS-paths (our table had over 186,000 prefixes and a bit over 31,000 unique AS paths). In our case, the average memory consumption was a bit more than 244 bytes per BGP prefix.

Interestingly, show memory alloc total command revealed BGP was not the largest memory consumer; IP routing table and FIB table have consumed more memory than the BGP table.

Largest memory consumers during the Internet in global routing table test

PE-A#sh mem alloc total 
               Head   Total(b)    Used(b)    Free(b)  Lowest(b) Largest(b)
Processor  64FB4E00  402958764  193812292  209146472  209100168  201632172
      I/O   E000000   33554432    3126436   30427996   30419888   30417180
Transient  7D000000   16777216     136332   16640884   16640884   16640852

Allocator PC Summary for: Processor

    PC          Total   Count  Name
0x60D6E524   37931664    1154  IP primary NDB 
0x6026C57C   29317836     447  IPv4 Unicast ne
0x61B4A978   25585788     760  CEF: fib
0x60D6E030   15274128    1519  IP PRIMARY RDB 
0x6026C4C0   13511128     206  BGP IPv4 Unicas
0x606C2970   11215204      45  Init
0x61BF5E20    9966704     302  TAL: MTRIE n08
0x630CE374    5955036    2038  IP routing tabl
0x630CE344    5937504    2032  IP routing tabl
0x60214F14    5247040      80  BGP attr chunk
0x60213E58    2266348      90  BGP (2) attr
0x62C61A4C    2089156     253  Process Stack
0x626E4AC0    1937628       2  RMON packet pool
0x628454AC    1463756     857  *Packet Header*
0x62845504    1258296     152  *Packet Data*

Average per-route memory consumption, calculated as the difference between free memory before and after the test divided by the number of prefixes: ~820 bytes per prefix.

Internet in a VRF

For the second test, I configured EBGP session with bgp_simple in a VRF:

Internet in a VRF

ip vrf Internet
 rd 65000:1
 route-target export 65000:1
 route-target import 65000:1
!
interface FastEthernet0/0
 ip vrf forwarding Internet
 ip address 10.17.0.1 255.255.255.0
!
router bgp 65000
 bgp log-neighbor-changes
 !
 address-family ipv4 vrf Internet
  neighbor 10.17.0.2 remote-as 65100
  neighbor 10.17.0.2 activate
 exit-address-family

Here are the relevant printouts (this time bgp_simple managed to send 196,074 prefixes from the same file containing 200,000 prefixes):

Free memory prior to the Internet in a VRF test

PE-A#sh mem free
               Head   Total(b)    Used(b)    Free(b)  Lowest(b) Largest(b)
Processor  64FB4E00  402960204   41474912  361485292  361465196  352628556
      I/O   E000000   33554432    3137124   30417308   30417308   30414684
Transient  7D000000   16777216       5156   16772060   16772032   16772028

BGP memory consumption in the Internet in a VRF test

PE-A#sh ip bgp vpnv4 vrf Internet sum
BGP router identifier 172.16.0.1, local AS number 65000
BGP table version is 196075, main routing table version 196075
196074 network entries using 32940432 bytes of memory
196074 path entries using 12548736 bytes of memory
71878/35939 BGP path/bestpath attribute entries using 9775408 bytes of memory
32494 BGP AS-PATH entries using 1599258 bytes of memory
1 BGP extended community entries using 24 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 56863858 total bytes of memory
BGP activity 196074/0 prefixes, 196074/0 paths, scan interval 60 secs

Neighbor    V     AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.17.0.2   4  65100  196084       8   196075    0    0 00:02:14   196074

Largest memory consumers in the Internet in a VRF test

PE-A#sh mem alloc totals 
               Head   Total(b)    Used(b)    Free(b)  Lowest(b) Largest(b)
Processor  64FB4E00  402958604  215429140  187529464  187485516  184854796
      I/O   E000000   33554432    3128932   30425500   30417392   30414684
Transient  7D000000   16777216     136332   16640884   16640884   16640852

Allocator PC Summary for: Processor

    PC          Total   Count  Name
0x60D6E524   39820880    1212  IP primary NDB 
0x6026C57C   34696052     529  VPNv4 Unicast n
0x61B4A978   26805108     796  CEF: fib
0x60D6E030   16032940    1595  IP PRIMARY RDB 
0x6026C4C0   14167008     216  BGP VPNv4 path-
0x60214F14   11281136     172  BGP attr chunk
0x606C2970   11215204      45  Init
0x61BF5E20   10428632     316  TAL: MTRIE n08
0x630CE374    6323208    2164  IP routing tabl
0x630CE344    5972592    2044  IP routing tabl
0x60213E58    2338592      94  BGP (1) attr
0x602A77B4    2198940      67  BGP TX attr top
0x62C61A4C    2083104     252  Process Stack
0x626E4AC0    1937628       2  RMON packet pool
0x628454AC    1468880     860  *Packet Header*
0x62845504    1258296     152  *Packet Data*
0x6021486C    1068624     349  BGP attrlist-ch
0x602158A8    1063712     348  BGP attrlist-ch

Average memory consumption:

  • ~290 bytes per BGP table entry (increase of 45 bytes per prefix or 18%)
  • ~890 bytes total per IP prefix (increase of ~70 bytes per prefix or 8%)

Internet in a VRF with MPLS/VPN route replication

Finally, I configured another VRF on the same PE-router and used the overlapping VPN design (different route distinguishers and common route target) to leak Internet routes into a customer VRF:

Additional configuration on the PE-router

PE-A#sh run vrf Customer
Building configuration...

Current configuration : 95 bytes
ip vrf Customer
 rd 65000:2
 route-target export 65000:1
 route-target import 65000:1

As expected, both BGP table size and total memory consumption increased significantly due to duplicate entries in the BGP table and duplicate FIBs (each VRF has its own FIB).

BGP memory consumption with route leaking between Internet and Customer VRF

PE-A#sh ip bgp vpnv4 all sum
BGP router identifier 172.16.0.1, local AS number 65000
BGP table version is 392149, main routing table version 392149
392148 network entries using 65880864 bytes of memory
392148 path entries using 25097472 bytes of memory
71878/35939 BGP path/bestpath attribute entries using 9775408 bytes of memory
32494 BGP AS-PATH entries using 1599258 bytes of memory
1 BGP extended community entries using 24 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 102353026 total bytes of memory
BGP activity 392148/0 prefixes, 392148/0 paths, scan interval 60 secs

Neighbor    V     AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.17.0.2   4  65100  196100      26   392149    0    0 00:07:40   196074

Total memory consumption with route leaking between Internet and Customer VRF

PE-A#sh mem alloc total
                Head  Total(b)    Used(b)    Free(b)  Lowest(b) Largest(b)
Processor  64FB4E00  402957004  374350796   28606208   28597336   17081036
      I/O   E000000   33554432    3128932   30425500   30417392   30414684
Transient  7D000000   16777216     136332   16640884   16640884   16640852

Allocator PC Summary for: Processor

    PC          Total   Count  Name
0x60D6E524   79605176    2424  IP primary NDB 
0x6026C57C   69392104    1058  VPNv4 Unicast n
0x61B4A978   53562408    1586  CEF: fib
0x60D6E030   32053992    3188  IP PRIMARY RDB 
0x6026C4C0   28334016     432  BGP VPNv4 path-
0x61BF5E20   20923268     634  TAL: MTRIE n08
0x630CE344   12243204    4190  IP routing tabl
0x630CE374   12219836    4182  IP routing tabl
0x60214F14   11281136     172  BGP attr chunk
0x606C2970   11215204      45  Init
0x602A77B4    4365060     133  BGP TX attr top
0x60213E58    2338592      94  BGP (1) attr
0x62C61A4C    2083104     252  Process Stack
0x626E4AC0    1937628       2  RMON packet pool
0x630CE1F0    1700604     582  IP routing tabl
0x628454AC    1468880     860  *Packet Header*
0x62845504    1258296     152  *Packet Data*
0x6021486C    1068624     349  BGP attrlist-ch
0x602158A8    1063712     348  BGP attrlist-ch

Average memory consumption with two VRFs carrying Internet routes:

  • ~520 bytes per BGP table entry (increase of ~230 bytes or ~80%)
  • Total ~1700 bytes per IP prefix (increase of ~810 bytes per prefix or over 90%)

15 comments:

  1. Great article as always.

    So here is my summary in short to see if I do understood it right

    So Internet in a vrf has a 8% /18% more Memory usage.
    This memory usage is Quite Huge ~80% if you import into another VPN instance.


    What I am missing in this comparissing is following setup:
    Internet in the Global table
    Import Global table into vrf.

    ReplyDelete
    Replies
    1. You got it right.

      If you import Internet routes from global BGP into a VRF, you'd probably still have a single copy of the BGP table (otherwise it would be identical to the inter-VRF import scenario), but two copies of RIB (global+VRF) and FIB (global+VRF).

      There's enough data in the above printouts to estimate what the increase in memory consumption would be; I'd guess it would be closer to 80% than to 8%.

      Delete
  2. What hardware platform? Sorry if I missed it in there. The platform I was choking on was the ASR 1K. In addition to the RIB, BGP topology info, and CEF, all in the IOS container, it makes a copy of the CEF data structure in main memory as well.

    ReplyDelete
  3. Posts like these actually bring in more interest @ ipspace.net :) Datacenter posts goes beyond my capability :P

    ReplyDelete
  4. Given that this just consumes additional RAM (not actual TCAM for packet forwarding), and that most new edge devices have somewhere from 4g to 16g to play with... is this a real problem on a modern network? RAM is something we can throw at problems to make them go away, large production x86 boxes often have hundreds of gig of ram.

    Alternatively, what are the advantages to keeping complete control and isolation of multiple upstream internet connections?

    Certainly the ability to control import and export PER upstream peer open the door to new revenue opportunities. Imagine you had a dozen peers (each with its own internet RT) to level3, cogent, tata, xo, zayo, etc. You would be able to sell a custom blend mix of BGP per customer by importing those upstream provider RT's into their internet VPN giving them the desired blended billing rate.

    Regional routing control could be more granular on a single-AS backbone (only import in-region peered providers into a VRF), VERY fast repair time with prefix independant convergence, and potentially some useful loadsharing options add themselves to the tool belt as well.

    This is a long standing, heated, design discussion topic between customers and vendors. The impact is fairly clear, but the actual cost of burning memory vs potential feature benefit later are murky.

    Right now my approach is splitting the best of both worlds by keeping the private IP VPN service on one set of edge devices, and the internet peering on another. If an internet VRF on the VPN PE needs to get out, I provide it a default route from the internet PE in the same POP (Different BGP ASN) and let it make the intelligent forwarding decisions based on a giant internet global routing table.

    That all said, I might rig up a test of my own with some of the new asr9001's at work. I am more than a little curious to see what happens with more upstream peers.

    ReplyDelete
    Replies
    1. Actually, we were discussing ideas along the same lines at the time when I was writing the MPLS/VPN book, the problem always being the amount of memory you'd need for numerous copies of full Internet routing table.

      Also, it's not just RAM. Every RIB (global and VRF) gets copied into FIB which gets copied into TCAM or equivalent forwarding structure.

      If you have two copies of full Internet routing table (one in global RIB, one in a VRF, for example), you're using twice as much TCAM as well.

      Delete
    2. One last side thought, would this help: www.asiafi.net/meeting/2011/HKWorkshop/Robert%20Raszuk.pdf

      Delete
    3. ILNPv6? It just might ... assuming you can change TCP stacks in all the hosts connected to the Internet in reasonable time.

      On the other hand, there just might be a gradual migration path (non-ILNPv6 hosts wouldn't get the extra resilience), but nobody would want to be the first one to go down it.

      Delete
  5. You are correct, my statement regarding TCAM was poorly worded. I was trying to reference that instanciating VRFs on the peering edge devices for customers was not necessary, and vice versa on the customer edge devices. I will have to think that all through again and see if there is a workaround to scaling the FIB.

    ReplyDelete
    Replies
    1. If you're leaking Internet routes from one VRF to another, you can avoid the TCAM hit (because you're only instantiating customer VRFs on the customer-facing PE-routers).

      If you carry Internet routing in the global routing table, then you'll always get double TCAM entries.

      Delete
  6. Maybe you should mention MPLS VPN Per VRF Label.

    http://www.cisco.com/en/US/docs/ios-xml/ios/mp_l3_vpns/configuration/15-0s/mp-vpn-vrf-label.html

    ReplyDelete
    Replies
    1. Per-VRF-label would reduce just the LFIB table size ... and I can't find LFIB in the list of top memory consumers. Am I missing something?

      Welcome back! Long time no hear.
      Ivan

      Delete
  7. Hi Ivan,

    Love the description of this issue, very informative. Are there any case-studies you've come cross where the MPLS Provider is providing both L3VPN services and Internet services in a single VRF, and sending more than a default-route for Internet services into the VRF?

    Thanks!

    ReplyDelete
  8. I have seen some providers offer internet services in vrf, but that is just 1 global vrf for customers and separate vrf per customer for private services. This does scale when just 1 internet vrf is used and internet peering is done directly in the vrf. But you loss and can be waiting for functionality to be implement on vrf interface/service which is only available today in the grt.

    ReplyDelete

You don't have to log in to post a comment, but please do provide your real name/URL. Anonymous comments might get deleted.

Ivan Pepelnjak, CCIE#1354, is the chief technology advisor for NIL Data Communications. He has been designing and implementing large-scale data communications networks as well as teaching and writing books about advanced technologies since 1990. See his full profile, contact him or follow @ioshints on Twitter.