Duplicate ARP Replies with Anycast Gateways
A reader sent me the following intriguing question:
I’m trying to understand the ARP behavior with SVI interface configured with anycast gateways of leaf switches, and with distributed anycast gateways configured across the leaf nodes in VXLAN scenario.
Without going into too many details, the core dilemma is: will the ARP request get flooded, and will we get multiple ARP replies. As always, the correct answer is “it depends” 🤷♂️
MUST READ: ARP Problems in EVPN
Decades ago there was a trick question on the CCIE exam exploring the intricate relationships between MAC and ARP table. I always understood the explanation for about 10 minutes and then I was back to I knew why that’s true, but now I lost it.
Fast forward 20 years, and we’re still seeing the same challenges, this time in EVPN networks using in-subnet proxy ARP. For more details, read the excellent ARP problems in EVPN article by Dmytro Shypovalov (I understood the problem after reading the article, and now it’s all a blur 🤷♂️).
Directed ARP Saga Continues
Reading my Directed ARP and ICMP Redirects blog post you might have wondered “how did Directed ARP ever get into ***redacted***?”
I searched for “directed ARP cisco” and found this gem, which really talks about unicast ARP behavior, an ancient mechanism documented in RFC 1122 (it’s not my Google-Fu, I got the reference to RFC 1122 in this blog post).
Directed ARP and ICMP Redirects
One of my readers sent me this question:
When I did my ***redacted*** I encountered a question about Directed ARP. The RFC (https://tools.ietf.org/html/rfc1433) is in the "experimental" stage, and I found it really weird from ***** to include such a hidden gem in the ***redacted***.
Directed ARP is clearly one of those weird things that people were trying out in the early days of networking when packet forwarding and bandwidth were still expensive (read the RFC for more details), but I kept wondering “what exactly is going on when a host receives an ICMP redirect?” Time for a hands-on test.
ARP Processing in Layer-3-Only Networks
John Jackson wrote an interesting comment on my Rearchitecting L3-Only Networks blog post:
What the host has configured for its default gateway doesn't really matter, correct? Because the default gateway in traditional L2 access networks really isn't about the gateway's IP address, but the gateway's MAC address. The destination IP address in the packet header is always the end destination IP address, never the default gateway.
He totally got the idea, however there are a few minor details to consider.
Optimal Layer-3 Forwarding with Active/Active VRRP (Enterasys Fabric Routing)
Enterasys implemented optimal layer-3 forwarding with an interesting trick: they support VRRP like any other switch vendor, but allow you to make all members of a VRRP group active forwarders regardless of their status.
Apart from a slightly more synchronized behavior, their implementation doesn’t differ much from Arista’s Virtual ARP, and thus shares the same design and deployment caveats.
For more information, watch the Fabric Routing video from the Enterasys Robust Data Center Interconnect Solutions webinar.
Arista EOS Virtual ARP (VARP) Behind the Scenes
In the Optimal L3 Forwarding with VARP and Active/Active VRRP blog post I made a remark along the lines of “Things might get nasty [in Arista EOS Virtual ARP world] if you have configuration mismatches”, resulting in a lengthy and amazingly insightful email exchange with Lincoln Dale during which we ventured deeper and deeper down the Virtual ARP (VARP) rabbit hole. Here’s what I learned during out trip:
Optimal L3 Forwarding with VARP and Active/Active VRRP
I’ve blogged about the need for optimal L3 forwarding across the whole data center in 2012 when I introduced it as one of the interesting requirements in Data Center Fabrics webinar. Years later, the concept became one of the cornerstones of modern EVPN fabrics, but there are still only a few companies that can deliver this functionality in a more traditional environment.
Mobile ARP in Enterprise Networks
Keith sent me a set of Mobile ARP questions, starting with “What’s your view on using Mobile ARP in a large enterprise?”
Short summary: Mobile ARP is an ancient technology that was designed to solve a problem that disappeared with deployment of DHCP. Now let’s look at the bigger picture.
Follow-up: Interface Default Route
Judging by your comments, some of you have already faced a stupidity similar to the one I’ve described on Friday. The symptoms are well described in the comments: the CPU utilization of the ARP process increases, packet forwarding becomes sluggish and the router runs out of memory, potentially resulting in a router crash. Now let’s analyze what’s going on.
My Stupid Moments: Interface Default Route
Years ago I was faced with an interesting challenge: an Internet customer was connected to our PE router with an Ethernet link and I did not want to include the PE router’s IP address in the default route on the CE router.
After pondering the problem for a while, I got a “brilliant” idea: if I would use an interface default route, proxy-ARP would solve all my problems. This is the configuration I’ve deployed on the CE-router:
Generating layer-2 broadcast from a regular IP packet
The Wake-on-LAN discussion we had a while ago brought us nowhere; there's simply no way to generate UDP packets on the router. I thought I could use Application Performance Monitor's Tcl scripts to generate the packet, but it looks like APM has been removed from recent IOS releases (and it's not clear whether you can use APM without a peer router).
The discussion nonetheless had an interesting side effect. Robert Turnšek sent me an interesting trick: with static ARP you can generate layer-2 broadcasts with a layer-3 unicast packet.
ARP timeout resolution is implemented in minutes
ARP entries are periodically refreshed if you use CEF switching
In a previous post I've been writing about the inability to clean the ARP cache due to cached CEF adjacencies. As it turns out, this behavior has another side effect: the router will automatically refresh all ARP entries (and CEF adjacencies) as they expire from the ARP cache. This might become a problem on high-end devices with a lot of directly connected hosts if you set the arp timeout to a low value.
What is a cached CEF adjacency?
Whenever a router running CEF switching has LAN interfaces (or any other multi-access interfaces), you'll find cached adjacencies for active directly attached IP neighbors in its CEF table. These adjacencies ensure the smooth traffic flow toward the LAN-attached next-hops (preventing the initial packet drop symptom once the next-hop becomes active).