Building network automation solutions

9 module online course

Start now!

Unnumbered Ethernet Interfaces

Imagine an Internet Service Provider offering Ethernet-based Internet access (aka everyone using fiber access, excluding people believing in Russian dolls). If they know how to spell security, they might be nervous about connecting numerous customers to the same multi-access network, but it seems they have only two ways to solve this challenge:

  • Use private VLANs with proxy ARP on the head-end router, forcing the customer-to-customer traffic to pass through layer-3 forwarding on the head-end router.
  • Use a separate routed interface with each customer, wasting three-quarters of their available IPv4 address space.

Is there a third option? Can’t we pretend Ethernet works in almost the same way as dialup and use unnumbered IPv4 interfaces?

Considering what we already know about unnumbered interfaces, it shouldn’t be too hard to get it done, considering that Ethernet forces you to supply next-hop layer-2 information even on point-to-point links.

Here’s what we have to do:

  • We’d need a static host route on our access router for each client, pointing toward an unnumbered interface. The static routes could be preconfigured or created on the fly based on DHCP address allocation.
  • We’d need ARP entries for client IP addresses connected to unnumbered interfaces to get the destination MAC address for the outbound IP packets.

It turns out that we had all the necessary bits and pieces for decades but couldn’t use them because network operating systems didn’t allow us to configure unnumbered multi-access interfaces.

Recent versions of Cisco IOS XE, Cisco Nexus OS, Arista EOS, and Junos no longer have that restriction. If anyone feels like tracking down the exact software version that added unnumbered Ethernet functionality, please write a comment.

Using Static Routes

We’ll start with the simplest scenario: using static routes. We’ll use a simple lab with an access router and two clients, each one of them connected to the access router with an Ethernet point-to-point link:

Lab topology

Lab topology

First, we need to configure a few unnumbered interfaces on the access router.

Unnumbered Ethernet interfaces on access router
interface Loopback0
 ip address 10.2.1.1 255.255.255.255
!
interface GigabitEthernet2
 description rtr -> c1
 ip unnumbered Loopback0
!
interface GigabitEthernet3
 description rtr -> c2
 ip unnumbered Loopback0

Next, we configure static routes for the client IP addresses pointing to those interfaces:

Static routes pointing toward the clients
ip route 10.2.1.2 255.255.255.255 GigabitEthernet2
ip route 10.2.1.3 255.255.255.255 GigabitEthernet3

That’s it. Now you can log in to one of the clients and ping the other client.

I’m ignoring a dirty detail – you have to configure IP addresses on clients' uplinks. See device configurations for details.

How Does It Work?

Here’s the magic trick: whenever you configure a static route pointing to an interface (without a next-hop), the router assumes that any destination within the configured prefix is directly connected (even though all printouts claim it’s still a static route).

Routing table on the access router
rtr#show ip route 10.2.0.0 longer-prefixes | begin Gateway
Gateway of last resort is 192.168.121.1 to network 0.0.0.0

      10.0.0.0/32 is subnetted, 3 subnets
C        10.2.1.1 is directly connected, Loopback0
S        10.2.1.2 is directly connected, GigabitEthernet2
S        10.2.1.3 is directly connected, GigabitEthernet3
rtr#show ip route 10.2.1.2
Routing entry for 10.2.1.2/32
  Known via "static", distance 1, metric 0 (connected)
  Routing Descriptor Blocks:
  * directly connected, via GigabitEthernet2
      Route metric is 0, traffic share count is 1

How do you reach a directly connected IP host? You ARP for its IP address and use the MAC address you get back to send the packet to it… and it doesn’t matter if the destination IP address belongs to the same subnet or not. ARP will kick in as soon as a router forwards an IP packet using a route without a next-hop.

Here’s the (expected) ARP cache on our access router. The loopback IP address has an ARP entry on all unnumbered interfaces; the ARP entries for adjacent IP addresses (10.2.1.2 and 10.2.1.34) point to different interfaces.

ARP cache on the access router
rtr#show arp 10.2.0.0 255.255.0.0
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.2.1.1                -   5254.0049.6b64  ARPA   GigabitEthernet2
Internet  10.2.1.1                -   5254.003a.5bba  ARPA   GigabitEthernet3
Internet  10.2.1.2                1   5254.00e3.3048  ARPA   GigabitEthernet2
Internet  10.2.1.3                1   5254.00cb.bf02  ARPA   GigabitEthernet3

Mission accomplished? Not exactly. While it’s possible to automate static route configuration, we still need to tell the clients what IP addresses to use. Time to add DHCP to the picture – the topic of the next blog post in this series.

Want to Know More?

  • The configurations I used to generate the printouts are on GitHub.
  • I discussed numerous aspects of network addressing in How Networks Really Work webinar.

Blog posts in this series

9 comments:

  1. How would it behave if you would use the IOS command "medium p2p" on the ethernet interfaces? How would the ARP table look like? What MAC addresses would be used to send packets?

  2. Dear Anonymous,

    The "medium p2p" interface configuration command is used on NX-OS. I didn't have to use it on Cisco IOS XE... as you would have discovered by looking at the device configurations (there's a reason I added a link to them at the end of the blog post).

    As for "how would the printouts look like" - I don't have them handy, and I don't have time to collect them, but here's what you can do: set up a lab, and do the tests yourself. Nexus 9300v is downloadable as a Vagrant box, and you can use my topology file with netsim-tools, changing the default device type from csr to nxos.

    Here's the documentation to get you started https://netsim-tools.readthedocs.io/en/latest/

    Once you get those printouts, do get in touch and I'll add them to the blog post.

    Thank you! Ivan

  3. Cisco , Juniper etc have support for unnumbered ethernet style services for a long time. It was just wrapped up in the Broadband Subscriber services functions where they could charge extra for it and dynamic in nature.

    It was long enough ago I was doing it on Cisco 7200s.

  4. When I was testing it on a Cisco 7200 long long time ago, it only worked on VLAN subinterfaces (not on the main interface), and not with routing protocols... but yeah, that was exactly what we needed for Broadband Subscriber services.

  5. Hi,

    Really good to see your article on this. We have been using this in production for about 2 years now on the Cisco 4500x platform in vss.

    The upside to all this is that migrating clients from a shared vlan to dedicated vlans and nothing from the client end routing or subletting had to change.

    Of course we only use static routing here. The clients can not accidentally start arping for another client ip or take over the default gateway arp accidentally or intentionally and affect other clients.

    The other plus is that we can apply per vlan policers in and out per client by vlan and not by IP.

  6. We started to use this setup in our out-of-band network where production devices have their fxp0, em0 and what not interfaces connected (it's all Juniper based). Initially, OoB network was just a flat switched network, but then one day due to human error we had forwarding loop there and ... let's just say control plane of almost all devices stopped running the actual network as it was too busy to process lovely flood of ARP packets. It wasn't pretty. But back to unnumbered interfaces. We use EX3400 with QFX5100 for some aggregation in this out-of-band network. As it was mentioned, the best part is that from connected device's perspective, nothing has changed. We've enabled also proxy-arp on these unnumbered interfaces. For downstream routing it's just static routes pointing to each downstream interface through qualified-next-hop and for upstream routing (each access EX3400 is dual uplinked for redundancy) we use venerable RIP protocol. Uplink interfaces are actually configured as aggregated ethernet interfaces with fast LACP and just single member. That is in order to make not so obvious link failures fast for RIP. Of course, one can be a license and use something else than RIP and maybe even VRFs if the same network serves multiple tenants, but for us this is good enough. It's somewhat less elegant when we are connecting dual control plane devices, where each routing engine has its own IP address, but they also share third one. In this case, that third IP address must be able to move between two physical ports. And IRB interface can't be configured as unnumbered interface. Until we find better solution, we create local vlan dedicated for a given two switched downstream ports and loop it over two front ports (one is configured as vlan trunk and another as L3 interface with sub-interfaces). Three static routes then point to the L3 sub-interface. Also to make it less painful to configure, we have commit script which transforms macro applied at interface level to all necessary statements in various places (the actual interface configuration, static route(s) and for dual RE case also vlan, new sub-interface, etc.).

  7. Arista Networks EOS 4.23.0F introduced support for IPv4 unnumbered interfaces. EOS 4.26.0.1F is the latest version as of this comment.

    https://eos.arista.com/eos-4-23-0f/ipv4-unnumbered-interfaces/

  8. Hello Ivan,

    I tried a similar configuration in order to use three devices in a /30 segment. .1 and .2 devices where already running, my goal was to use the .0 (i am completly aware that using the .3 address was not a good idea). I used your configuration :

    interface Loopback2412
      ip address 10.10.10.0 255.255.255.255
    interface GigabitEthernet0/1.2412
      encapsulation dot1Q 2412
      ip unnumbered Loopback2412
    ip route 10.10.10.0 255.255.255.252 GigabitEthernet0/1.2412
    

    whereas the two other devices on the subnet have a regular configuration:

    lyo-maq-2901-08#sh run int gi 0/1.2412
    interface GigabitEthernet0/1.2412
      encapsulation dot1Q 2412
      ip address 10.10.10.1 255.255.255.252
    

    I was able to arp the neighbours, but unfortunately not to ping them, since the neighbours do not answer even if they correctly learn the arp entry :

    lyo-maq-2901-08#sh arp
    Protocol  Address          Age (min)  Hardware Addr   Type   Interface
    Internet  10.10.10.0              8   70ca.9bae.f8b1  ARPA   GigabitEthernet0/1.2412
    

    I think the reason is to find inside the RFC 1812: > a router [...] (2) SHOULD silently discard on receipt (i.e., do not even deliver to applications in the router) any packet addressed to 0.0.0.0 or { <Network-prefix>, 0 }. If these packets are not silently discarded, they MUST be treated as IP broadcasts (see Section [5.3.5]). There MAY be a configuration option to allow receipt of these packets. This option SHOULD default to discarding them.

    which matchs somehow the cef status on the neighbour :

    lyo-maq-2901-08#sh cef
    10.10.10.0/30        attached             GigabitEthernet0/1.2412
    10.10.10.0/32        receive              GigabitEthernet0/1.2412
    10.10.10.1/32        receive              GigabitEthernet0/1.2412
    10.10.10.3/32        receive              GigabitEthernet0/1.2412
    

    Eventually, this can be bypassed by configuring the neighbours with the unnumbered trick :

    lyo-maq-2901-08#conf t
    Enter configuration commands, one per line.  End with CNTL/Z.
    lyo-maq-2901-08(config)#int lo 2412
    lyo-maq-2901-08(config-if)#ip add 10.10.10.1 255.255.255.255
    lyo-maq-2901-08(config-if)#int gi 0/1.2412
    lyo-maq-2901-08(config-subif)#no ip addr
    lyo-maq-2901-08(config-subif)#ip unnumbered loopback 2412
    lyo-maq-2901-08(config-subif)#ip route 10.10.10.0 255.255.255.252 gi 0/1.2412
    lyo-maq-2901-08(config)#end
    lyo-maq-2901-08#sh ip cef
    Prefix               Next Hop             Interface
    10.10.10.0/30        attached             GigabitEthernet0/1.2412
    10.10.10.0/32        attached             GigabitEthernet0/1.2412
    10.10.10.1/32        receive              Loopback2412
    

    Lab done, but not sure i want to deploy it :)

    lyo-maq-2901-09#ping 10.10.10.1
    Type escape sequence to abort.
    Sending 5, 100-byte ICMP Echos to 10.10.10.1, timeout is 2 seconds:
    !!!!!
    Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
    lyo-maq-2901-09#
    
  9. Note that on IOS & IOS-XE you can't use static routes pointing to interface without next-hop inside a VRF :

    ip route vrf TEST 10.2.1.2 255.255.255.255 GigabitEthernet2

    will raise an error : % For VPN or topology routes, must specify a next hop IP address if not a point-to-point interface

    You can overcome this, either by configuring the interfaces with "medium p2p" (not available on classic IOS) or by using your route destination as next-hop :

    ip route vrf TEST 10.2.1.2 255.255.255.255 GigabitEthernet2 10.2.1.2

    which is accepted and seems to work perfectly... but it's weird and recursive :)

Add comment
Sidebar