4

Linux server has 2 active network interfaces:

IF:eth1    IP:192.168.1.1/24    MAC:11:11:11:11:11:11   (1GbE)
IF:eth2    IP:192.168.1.2/24    MAC:22:22:22:22:22:22   (10GbE)

The idea is that the 10GbE interface (eth2) is the primary interface for communication with hosts on the network. I want to leave the second 1GbE interface (eth1) up as a failsafe. In the event that the 10GbE interface goes down: I'd still have an easy way in, can update DNS so hosts can connect, etc.

While observing interface statistics I noticed that all traffic was sending/receiving on the eth1 instead of eth2 despite the fact that all hosts on the network are addressing this interface. I confirmed DNS A record points to the IP of the correct interface. Additionally, I confirmed that addressing the interface by IP instead of FQDN produces the same result.

I cleared the ARP cache on my machine and pinged the eth1 interface by IP address. I inspect my ARP table and find the MAC address of eth1. I cleared the ARP cache again and pinged the eth2 interface by IP address. Again, I inspect my ARP table and find the MAC address of eth1 (not eth2).

If I bring down eth1, physically disconnect the interface, or put it on a different logical network - I get the expected behavior, traffic goes over my eth2 interface.

My question: Why does this happen? I am seeing some evidence that this is expected behavior on the linux kernel due to its "weak host model."

How can I keep both interfaces up, on the same network, and have them work in the way that I expect.

sardean
  • 853

2 Answers2

3

Linux is designed to respond to ARP requests on any interface. It is assumed that the host owns the IP address and not the particular interface. What you are seeing is called ARP Flux.

You can change this behavior using sysctrl

arp_ignore - INTEGER

Define different modes for sending replies in response to received ARP requests that resolve local target IP addresses:

0 - (default): reply for any local target IP address, configured on any interface

1 - reply only if the target IP address is local address configured on the incoming interface

2 - reply only if the target IP address is local address configured on the incoming interface and both with the sender's IP address are part from same subnet on this interface

3 - do not reply for local addresses configured with scope host, only resolutions for global and link addresses are replied

David Houde
  • 3,240
3

If your switch supports it, I would use 802.1ad link aggregation for providing failover.

With this feature, you bond the two interfaces together, and you can set one as active and one as passive interface. Your IP address would reside on the bonding interface, so there would be no IP address changes if one NIC fails.

Tero Kilkanen
  • 38,887