Category: ARP

The amount of weird stuff we discover in netlab integration tests is astounding, or maybe I have a knack for looking into the wrong dark corners (my wife would definitely agree with that). Today’s special: when having two next hops kills a static route.

TL&DR: default ARP settings on a multi-subnet Linux host are less than optimal.

We use these principles when creating netlab integration tests:

• They should contain a single device-under-test and a bunch of attached probes.
• They should test a single feature.
• They should not rely on the device-under-test. All validation has to be done on probes.

How do you test static routes under these restrictions? Here’s what we did:

After figuring out ARP details, describing how routers use ARP to resolve entries in the IP routing table, and considering what we already know about OSPF on unnumbered IPv4 interfaces, we’re finally ready to answer Daniel’s question:

A few days ago, I described how ARP behaves when the source- and destination IP addresses are not on the same subnet (TL&DR: it doesn’t care). Now, let’s see how routers use ARP to get the destination MAC address for various entries in the IP routing table. To keep things simple, we’ll use static routes to insert entries in the IP routing table.

We’ll run our tests in a small virtual lab with two Linux hosts and an Arista vEOS switch. The link between H1 and RTR is a regular subnet. H2 has an IP address on the Ethernet interface, but RTR uses an unnumbered interface.

When figuring out how unnumbered IPv4 interfaces work, Daniel Dib asked an interesting question: How does ARP work when the source and destination IPv4 address are not in the same segment (as is usually the case when using unnumbered interfaces)?

TL&DR: ARP doesn’t care about subnets. If the TCP/IP stack needs to find a MAC address of a node it thinks is adjacent, ARP does its best, no matter what.

A reader sent me the following intriguing question:

I’m trying to understand the ARP behavior with SVI interface configured with anycast gateways of leaf switches, and with distributed anycast gateways configured across the leaf nodes in VXLAN scenario.

Without going into too many details, the core dilemma is: will the ARP request get flooded, and will we get multiple ARP replies. As always, the correct answer is “it depends” 🤷‍♂️

Decades ago there was a trick question on the CCIE exam exploring the intricate relationships between MAC and ARP table. I always understood the explanation for about 10 minutes and then I was back to I knew why that’s true, but now I lost it.

Fast forward 20 years, and we’re still seeing the same challenges, this time in EVPN networks using in-subnet proxy ARP. For more details, read the excellent ARP problems in EVPN article by Dmytro Shypovalov (I understood the problem after reading the article, and now it’s all a blur 🤷‍♂️).

Reading my Directed ARP and ICMP Redirects blog post you might have wondered “how did Directed ARP ever get into ***redacted***?”

I searched for “directed ARP cisco” and found this gem, which really talks about unicast ARP behavior, an ancient mechanism documented in RFC 1122 (it’s not my Google-Fu, I got the reference to RFC 1122 in this blog post).

One of my readers sent me this question:

When I did my ***redacted*** I encountered a question about Directed ARP. The RFC (https://tools.ietf.org/html/rfc1433) is in the "experimental" stage, and I found it really weird from ***** to include such a hidden gem in the ***redacted***.

Directed ARP is clearly one of those weird things that people were trying out in the early days of networking when packet forwarding and bandwidth were still expensive (read the RFC for more details), but I kept wondering “what exactly is going on when a host receives an ICMP redirect?” Time for a hands-on test.

John Jackson wrote an interesting comment on my Rearchitecting L3-Only Networks blog post:

What the host has configured for its default gateway doesn't really matter, correct? Because the default gateway in traditional L2 access networks really isn't about the gateway's IP address, but the gateway's MAC address. The destination IP address in the packet header is always the end destination IP address, never the default gateway.

He totally got the idea, however there are a few minor details to consider.

Enterasys implemented optimal layer-3 forwarding with an interesting trick: they support VRRP like any other switch vendor, but allow you to make all members of a VRRP group active forwarders regardless of their status.

Apart from a slightly more synchronized behavior, their implementation doesn’t differ much from Arista’s Virtual ARP, and thus shares the same design and deployment caveats.

For more information, watch the Fabric Routing video from the Enterasys Robust Data Center Interconnect Solutions webinar.