Multihomed IP Hosts

IP host (workstations, servers or communication equipment) is multihomed if it has more than one IP address. An IP host can be multihomed in numerous ways, using one or more layer-3 interfaces for network connectivity. Some multihoming scenarios are well understood and commonly used, while others (multiple physical layer-3 interfaces in the same IP subnet) could be hard to implement on common operating systems.

A host having a routable IP address on an external interface and on or more IP addresses on internal interfaces like the loopback interface is not considered multihomed.

Multihoming Scenarios

RFC 1122 (Requirements for Internet Hosts) documents three multihoming scenarios:

  • Multiple logical networks. A host connected to a single layer-2 networks with multiple IP subnets may have an IP address in each subnet (secondary IP addresses in Cisco IOS terminology).
Multiple logical networks

Multiple logical networks

  • Multiple logical hosts. A host has multiple IP addresses in the same IP subnet. These IP addresses might be configured on the same or different layer-2 interfaces.
Multiple logical hosts; two interfaces

Multiple logical hosts; two interfaces

Multiple logical hosts; single interface

Multiple logical hosts; single interface

  • Simple multihoming. A host is connected to multiple layer-2 networks (and thus multiple IP subnets) and has one (or more, see Multiple logical hosts) IP address in each IP subnet.
Simple IP host multihoming

Simple IP host multihoming

Multihomed IP hosts face two significant challenges:

  • Session endpoint addressing: which IP address is used as the source IP address in outgoing IP datagrams?
  • Datagram forwarding: which interface is used to send outbound IP datagrams?

These challenges are well documented in RFC 1122 (see also the discussion below). Some operating systems or add-on networking software (for example, stateful host firewalls) might impose additional limitations.

Session Endpoint Addressing

A multihomed host could use any one of its IP addresses as the source IP address of outgoing IP datagrams carrying TCP or UDP packets. To ensure that the remote host is able to match its side of a TCP or UDP session with the received IP datagram, RFC 1122 imposes the following rules on IP address selection:

  • All segments in the same TCP session must use the same IP address (section 4.2.3.7, TCP multihoming).
  • A reply to an incoming UDP packet should use the same IP address to which the incoming UDP packet has been sent (section 4.1.3.5, UDP multihoming).
  • Application can specify the IP address of the first packet in a session if the session is initiated from the multihomed host.
  • If the application does not specify the IP address of the first packet in a TCP or UDP session, the TCP/UDP stack must ask IP layer to select the source IP address for the outgoing datagram. The source IP address is usually the IP address of the outgoing interface (section 3.3.4.3, Choosing a Source Address).

A server running on a multihomed host should therefore use the destination IP address of the request packet as the source IP address of the reply packet. The IP address selection is automatic for TCP-based services (where the TCP stack enforces the correct IP address) and a UDP server should ensure the correct IP address is used in the reply packets.

On some operating systems, the client application on a multihomed host can specify the source IP address of the TCP or UDP session (socket library example).

Datagram Forwarding

RFC 1122 documents datagram forwarding to local subnets and datagram forwarding to default gateways (it also allows hosts to use static routes), but does not address the interface selection rules when a multihomed host has multiple default gateways or multiple interfaces in the same IP subnet (Multiple logical hosts). These decisions are implementation dependent and many IP stacks use the first-match rule, sending majority of the outbound datagrams through a single interface.

Some operating systems (or add-on networking software) allow the administrator to define load sharing or policy routing rules. For example, the Linux iproute2 package allows you to  configure a policy where the source IP address selects the outgoing interface

Multiple Logical Hosts Issues

Even though the multiple logical hosts design (a host with two interfaces in the same IP subnet) is documented in RFC 1122, you could experience significant roadblocks when trying to use multiple IP addresses from the same IP subnet on multiple physical interfaces with modern operating systems, including:

  • Interaction with host firewall. Default IP datagram forwarding mechanisms in the host operating system often result in asymmetrical routing. Stateful firewalls might drop asymmetrically forwarded packets.
  • Duplicate MAC addresses. An operating system that uses the same MAC address on multiple interfaces could cause significant problems in a switched LAN environment, as packets sourced from the same MAC address through two physical interfaces might confuse the LAN switches.
  • Proxy ARP replies. A host with two interfaces connected to the same layer-2 network receives two copies of each ARP request. The host might decide to answer both ARP requests (one of them due to the proxy ARP mechanism), resulting in suboptimal IP-to-MAC address mappings in remote hosts.
An intrusion detection system might interpret dual reply to an ARP request as an indication of ARP spoofing attack.
  • Link loss issues. If a multihomed host loses one of its attachments to the IP subnet, the IP address configured on the failed interface might still be active although its corresponding MAC address is no longer reachable. If you want to use a multiple logical hosts implementation in a high-availability design, you should deploy a load balancing device with server loss detection mechanism in front of a multihomed host.
Some operating systems avoid the link loss issues by supporting transfer of IP and MAC addresses between physical interfaces.

More Details

Latest blog posts in Site and Host Multihoming series

5 comments:

  1. I followed the conversation on NANOG with much pleasure, from both the pro- and the con-side of the argument, regardless of RFC-'legality'.

    If I recall correctly the argument started because of loadbalancing/high availability issues.

    It gets even more interesting if you allow the host to bridge between these interfaces and an HA virtual interface, although I think that bundeling (bonding) of the interfaces the 'cleanest' solution is. Ofcourse there are some problems if your aggeregate is spanned over multiple switches because of HA demands, which often forces the use of active/passive constructions.

    Luckily switch-cluster technology (SMLT, VSS etc.) is becoming available on enterprise-grade hardware, which overcomes these problems.

    Question regarding the future, will there be host-implementations of L2MP? Oppertunity or headache?
  2. Several of our clients "acomplished" to implement multihomed servers generating a kind of "assymetric routing" which generated "exessive unicast-flooding". In one case (paired with huge filetransfers) this resulted in periodical "brownouts" of there core. This "phenomenon" could/should be added to the issues section.
  3. @Michael: Gladly. Could you send me more information? You can use the form accessible through the "Contact" link on the top of the page if we haven't communicated before.
  4. There were several comments in the NANOG thread suggesting that exit interface and source IP address are necessarily correlated.

    This just isn't true.

    For connectionless protocols, and for the first (SYN) packet in a TCP session, the correlation *might* be there, but even that can be changed.

    Consider the "source-interface" set of IOS commands, which force the first packet in a session to ignore the exit interface.

    For a mid-session example, consider this: I can ssh into the external interface of my linksys router from inside my home network.

    The exit interface of the response packet is the internal one, but the source address is the external one.

    I'm suprised how many people got this wrong.
  5. @Anonymous: the misconceptions in the NANOG thread prompted me to write the Wiki article.

    If you go through the TCP/UDP portion of it, you'll see the exact correlation between the outgoing interface and source IP address (when and where it does exist).
Add comment
Sidebar