Decades ago there was a trick question on the CCIE exam exploring the intricate relationships between MAC and ARP table. I always understood the explanation for about 10 minutes and then I was back to I knew why that’s true, but now I lost it.
Fast forward 20 years, and we’re still seeing the same challenges, this time in EVPN networks using in-subnet proxy ARP. For more details, read the excellent ARP problems in EVPN article by Dmytro Shypovalov (I understood the problem after reading the article, and now it’s all a blur 🤷♂️).
Reading my Directed ARP and ICMP Redirects blog post you might have wondered “how did Directed ARP ever get into ***redacted***?”
I searched for “directed ARP cisco” and found this gem, which really talks about unicast ARP behavior, an ancient mechanism documented in RFC 1122 (it’s not my Google-Fu, I got the reference to RFC 1122 in this blog post).
One of my readers sent me this question:
When I did my ***redacted*** I encountered a question about Directed ARP. The RFC (https://tools.ietf.org/html/rfc1433) is in the "experimental" stage, and I found it really weird from ***** to include such a hidden gem in the ***redacted***.
Directed ARP is clearly one of those weird things that people were trying out in the early days of networking when packet forwarding and bandwidth were still expensive (read the RFC for more details), but I kept wondering “what exactly is going on when a host receives an ICMP redirect?” Time for a hands-on test.
John Jackson wrote an interesting comment on my Rearchitecting L3-Only Networks blog post:
What the host has configured for its default gateway doesn't really matter, correct? Because the default gateway in traditional L2 access networks really isn't about the gateway's IP address, but the gateway's MAC address. The destination IP address in the packet header is always the end destination IP address, never the default gateway.
He totally got the idea, however there are a few minor details to consider.
Enterasys implemented optimal layer-3 forwarding with an interesting trick: they support VRRP like any other switch vendor, but allow you to make all members of a VRRP group active forwarders regardless of their status.
For more information, watch the Fabric Routing video from the Enterasys Robust Data Center Interconnect Solutions webinar.
In the "Optimal L3 Forwarding with VARP and Active/Active VRRP" blog post I made a remark along the lines of "Things might get nasty [in Arista EOS Virtual ARP world] if you have configuration mismatches", resulting in a lengthy and amazingly insightful email exchange with Lincoln Dale during which we ventured deeper and deeper down the Virtual ARP (VARP) rabbit hole. Here's what I learned during out trip:
I’ve blogged about the need for optimal L3 forwarding across the whole data center almost a year ago when I introduced it as one of the interesting requirements in Data Center Fabrics webinar. A year later, there are still only a few companies that can deliver this functionality.
Keith sent me a set of Mobile ARP questions, starting with “What’s your view on using Mobile ARP in a large enterprise?”
Short summary: Mobile ARP is an ancient technology that was designed to solve a problem that disappeared with deployment of DHCP. Now let’s look at the bigger picture.
Judging by your comments, some of you have already faced a stupidity similar to the one I’ve described on Friday (BTW, I’ve remembered this particular debacle when receiving a Pingsta case invitation with very similar symptoms). The symptoms are well described in the comments: the CPU utilization of the ARP process increases, packet forwarding becomes sluggish and the router runs out of memory, potentially resulting in a router crash. Now let’s analyze what’s going on.
Years ago I was faced with an interesting challenge: an Internet customer was connected to our PE router with an Ethernet link and I did not want to include the PE router’s IP address in the default route on the CE router.
The latest IOS release in those days was probably somewhere around 11.x; none of the DHCP goodies were available.
After pondering the problem for a while, I got a brilliant idea: if I would use an interface default route, proxy-ARP would solve all my problems. This is the configuration I’ve deployed on the CE-router:
interface Ethernet 0
description Uplink to the ISP
ip address 10.0.1.2 255.255.255.0
ip route 0.0.0.0 0.0.0.0 ethernet 0
We tested this configuration in the middle of the night and it worked as expected. What do you think happened in the morning?
The Wake-on-LAN discussion we had a while ago brought us nowhere; there's simply no way to generate UDP packets on the router. I thought I could use Application Performance Monitor's Tcl scripts to generate the packet, but it looks like APM has been removed from recent IOS releases (and it's not clear whether you can use APM without a peer router).
The discussion nonetheless had an interesting side effect. Robert Turnšek sent me an interesting trick: with static ARP you can generate layer-2 broadcasts with a layer-3 unicast packet.
In a previous post I've been writing about the inability to clean the ARP cache due to cached CEF adjacencies. As it turns out, this behavior has another side effect: the router will automatically refresh all ARP entries (and CEF adjacencies) as they expire from the ARP cache. This might become a problem on high-end devices with a lot of directly connected hosts if you set the arp timeout to a low value.
Whenever a router running CEF switching has LAN interfaces (or any other multi-access interfaces), you'll find cached adjacencies for active directly attached IP neighbors in its CEF table. These adjacencies ensure the smooth traffic flow toward the LAN-attached next-hops (preventing the initial packet drop symptom once the next-hop becomes active).