DMVPN spoke NHRP behavior changed in IOS release 15.0M

In the good old days, we (thought we) knew how Phase 2 DMVPN works and what happens when the spoke-to-spoke session cannot be established. As I discovered when developing the lab configurations for my new DMVPN: New Features in IOS Release 15 webinar (register here), that behavior has forever changed (and not for the better) sometime in the 12.4T (or 15.0M) release. I blame the introduction of NAT awareness in IOS release 12.4(15)T, but it could be another totally unrelated change.

I've set up a small lab network (actually reused the lab configurations you get as part of the DMVPN - From Basics to Scalable Networks or DMVPN New Features webinars) shown in the following diagram to illustrate the change in the behavior. After OSPF populated the routing tables on all routers, I pinged the loopback interface of R3 from R2.

The old behavior

A spoke (R2) sends a NHRP request asking for other spoke’s IP address to the hub router. The hub router propagates the NHRP request to the other spoke (R3), which then tries to respond (and build a spoke-to-spoke IPsec tunnel).

If the spoke-to-spoke IPsec tunnel fails to materialize, the two spokes have incomplete (and thus unusable) NHRP entries and the traffic continues to flow over the hub router(s).

R2#sh ip nhrp 192.168.0.6
192.168.0.6/32, Tunnel0 created 00:00:44, expire 00:02:20
  Type: incomplete, Flags: negative
  Cache hits: 2
R3#sh ip nhrp 192.168.0.5
192.168.0.5/32, Tunnel0 created 00:01:02, expire 00:02:02
  Type: incomplete, Flags: negative
  Cache hits: 2

The new behavior

As before, a spoke (R2) sends a NHRP request asking for other spoke’s (R3) IP address to the hub router and the hub router propagates the request to the other spoke. At the same time, the sending spoke (R2) creates a fake NHRP entry pointing to the hub router. The fake NHRP entry has a fixed 3-minute expiration time which cannot be changed (with non-hidden commands, at least) and which does not depend on the configured NHRP timers.

The receiving spoke (R3) gleans its peer IP address and builds its own NHRP entry. If the spoke-to-spoke IPsec tunnel establishment fails to materialize, the sending spoke (R2) retains the fake NHRP entry (pointing to the hub); the receiving spoke (R3) has a regular but unusable NHRP entry in no socket state (because the IPsec tunnel is not there).

R2#sh ip nhrp 192.168.0.6
192.168.0.6/32 via 192.168.0.6
   Tunnel0 created 00:00:17, expire 00:02:47
   Type: dynamic, Flags: used temporary
   NBMA address: 10.0.7.17
R3#sh ip nhrp 192.168.0.5
192.168.0.5/32 via 192.168.0.5
   Tunnel1 created 00:00:29, expire 00:00:30
   Type: dynamic, Flags: router implicit
   NBMA address: 10.0.7.9
    (no-socket)

As you can see, the expiration time of the fake NHRP entry is 3 minutes, while the expiration time of the gleaned NHRP entry corresponds to the ip nhrp holdtime value.

Is this a problem?

Yes it is; it’s a huge problem in some redundant network designs. Read this blog post for more details; they are also explained in my DMVPN webinars.

8 comments:

  1. We have about 20 Routers in our DNMVPN network. After a certain point we noticed that the network became unstable. What we found that helped was implementing no ip cef. We also switched to eigrp.

    Right now we are running 12.4(25a).

    ReplyDelete
  2. I ran into a similar issue with a bug showing up in 12.4(18e) code with DMVPN. I believe what you will see is that it still thinks there is a route to the opposing network so traffic routes and black holes on spoke to spoke communications even through the tunnel is not up to allow the traffic actually route.

    This was something I haven't seen come into play in any 15.x routers, it was only the 12.4(18e) routers I saw it on.

    Then again maybe I a missing what you are trying to point out.

    ReplyDelete
  3. Looks like R2 will forward towards the hub, while R3 will drop all traffic returning or heading towards 192.168.0.5. IPSec will not be triggered on R3, looks like a big issue, but that's as far as I can see.. could be wrong.

    The suspense is killing me.. ;)

    ReplyDelete
  4. Ivan Pepelnjak30 April, 2011 15:56

    Actually R3 will forward all traffic toward the hub, as its NHRP entry for R2 is not operational. No problem there.

    ReplyDelete
  5. You mentioned this will be a huge problem in redundant network designs: when C1 (your hub) is going to fail, R2 won't be able to connect with R3 for at least 3 minutes (time for the fake entry to disappear from R2's NHRP table) even though R2 and R3 may have already registered with another "C2" hub. This is a problem I can see.

    I stumbled upon your blog quite recently and this is just an awesome source of info.

    Cheers!

    ReplyDelete
  6. johanne.doughe07 March, 2012 09:05

    I have a similar problem The symtems sound familar. I have one spoke router (A) that is connected to router (A1) that has an alternate link to another router (C1) which is behind the Hub (C) router. Spoke (B) sends traffic to the hub and then down the alternate path from the hub (C) to (C1) to (A1). The router (A1) send traffic through the spoke router (A) through the tunnel to spoke (B).

    ReplyDelete
  7. Does anyone know the solution to why the spoke will not create the proper entry. I have the propblem with a particular which always shows the connect to another peer with no-socket and routes traffic through the headend which causes high latency on the network. How can I get the direct connection to work?

    ReplyDelete

You don't have to log in to post a comment, but please do provide your real name/URL. Anonymous comments might get deleted.

Ivan Pepelnjak, CCIE#1354, is the chief technology advisor for NIL Data Communications. He has been designing and implementing large-scale data communications networks as well as teaching and writing books about advanced technologies since 1990. See his full profile, contact him or follow @ioshints on Twitter.