I am trying to setup a basic Unicast VxLAN between 3 Ubuntu 22.04 virtual servers (underlay IPs : 172.16.30.26/24, 27 and 28) running on VirtualBox. No firewalls are active. UFW is removed. After the setup, when I try to ping between the 3 overlay IPs (192.168.200.1/24, 2, 3) I see ARP request reach the other machines, but no response is generated from the destination machine and hence the source machines keep saying: Destination Host Unreachable and I get No Route to Host. All unicast VxLAN config guides for Linux on the net point to this setup should work. Looking at the routing table, routes look fine and underlay pings work fine. Here is the configuration of the 3 machines and display of ping and tcpdump between VM1 and VM3.
VM1
sudo ip link add vxlan0 type vxlan id 42 dev enp0s3 dstport 0
sudo bridge fdb append to 00:00:00:00:00:00 dst 172.16.30.27 dev vxlan0
sudo bridge fdb append to 00:00:00:00:00:00 dst 172.16.30.28 dev vxlan0
sudo ip addr add 192.168.200.1/24 dev vxlan0
sudo ip link set up dev vxlan0
ed@ubuntu-06:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:5a:4b:3e brd ff:ff:ff:ff:ff:ff
inet 172.16.30.26/24 brd 172.16.30.255 scope global noprefixroute enp0s3
valid_lft forever preferred_lft forever
3: vxlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 92:f9:2d:1a:59:6e brd ff:ff:ff:ff:ff:ff
inet 192.168.200.1/24 scope global vxlan0
valid_lft forever preferred_lft forever
inet6 fe80::90f9:2dff:fe1a:596e/64 scope link
valid_lft forever preferred_lft forever
ed@ubuntu-06:~$ ip route
default via 172.16.30.1 dev enp0s3 proto static metric 100
169.254.0.0/16 dev enp0s3 scope link metric 1000
172.16.30.0/24 dev enp0s3 proto kernel scope link src 172.16.30.26 metric 100
192.168.200.0/24 dev vxlan0 proto kernel scope link src 192.168.200.1
VM2
sudo ip link add vxlan0 type vxlan id 42 dev enp0s3 dstport 0
sudo bridge fdb append to 00:00:00:00:00:00 dst 172.16.30.26 dev vxlan0
sudo bridge fdb append to 00:00:00:00:00:00 dst 172.16.30.28 dev vxlan0
sudo ip addr add 192.168.200.2/24 dev vxlan0
sudo ip link set up dev vxlan0
ed@ubuntu-07:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:6c:70:b9 brd ff:ff:ff:ff:ff:ff
inet 172.16.30.27/24 brd 172.16.30.255 scope global noprefixroute enp0s3
valid_lft forever preferred_lft forever
3: vxlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 92:f9:2d:1a:59:6e brd ff:ff:ff:ff:ff:ff
inet 192.168.200.2/24 scope global vxlan0
valid_lft forever preferred_lft forever
inet6 fe80::90f9:2dff:fe1a:596e/64 scope link
valid_lft forever preferred_lft forever
ed@ubuntu-07:~$ ip route
default via 172.16.30.1 dev enp0s3 proto static metric 100
169.254.0.0/16 dev enp0s3 scope link metric 1000
172.16.30.0/24 dev enp0s3 proto kernel scope link src 172.16.30.27 metric 100
192.168.200.0/24 dev vxlan0 proto kernel scope link src 192.168.200.2
VM3
sudo ip link add vxlan0 type vxlan id 42 dev enp0s3 dstport 0
sudo bridge fdb append to 00:00:00:00:00:00 dst 172.16.30.27 dev vxlan0
sudo bridge fdb append to 00:00:00:00:00:00 dst 172.16.30.26 dev vxlan0
sudo ip addr add 192.168.200.3/24 dev vxlan0
sudo ip link set up dev vxlan0
ed@ubuntu-08:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:9e:e4:6c brd ff:ff:ff:ff:ff:ff
inet 172.16.30.28/24 brd 172.16.30.255 scope global noprefixroute enp0s3
valid_lft forever preferred_lft forever
3: vxlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 92:f9:2d:1a:59:6e brd ff:ff:ff:ff:ff:ff
inet 192.168.200.3/24 scope global vxlan0
valid_lft forever preferred_lft forever
inet6 fe80::90f9:2dff:fe1a:596e/64 scope link
valid_lft forever preferred_lft forever
ed@ubuntu-08:~$ ip route
default via 172.16.30.1 dev enp0s3 proto static metric 100
169.254.0.0/16 dev enp0s3 scope link metric 1000
172.16.30.0/24 dev enp0s3 proto kernel scope link src 172.16.30.28 metric 100
192.168.200.0/24 dev vxlan0 proto kernel scope link src 192.168.200.3
PING from VM1 Overlay to VM3 Overlay:
ed@ubuntu-06:~$ ping 192.168.200.3
PING 192.168.200.3 (192.168.200.3) 56(84) bytes of data.
From 192.168.200.1 icmp_seq=1 Destination Host Unreachable
ping: sendmsg: No route to host
From 192.168.200.1 icmp_seq=2 Destination Host Unreachable
From 192.168.200.1 icmp_seq=3 Destination Host Unreachable
From 192.168.200.1 icmp_seq=5 Destination Host Unreachable
ping: sendmsg: No route to host
From 192.168.200.1 icmp_seq=6 Destination Host Unreachable
From 192.168.200.1 icmp_seq=7 Destination Host Unreachable
^C
--- 192.168.200.3 ping statistics ---
11 packets transmitted, 0 received, +6 errors, 100% packet loss, time 10230ms
pipe 3
tcpdump on VM3 monitoring port 8472:
ed@ubuntu-08:~$ sudo tcpdump -ni enp0s3 port 8472
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp0s3, link-type EN10MB (Ethernet), snapshot length 262144 bytes
16:12:48.118239 IP 172.16.30.26.45506 > 172.16.30.28.8472: OTV, flags [I] (0x08), overlay 0, instance 42
ARP, Request who-has 192.168.200.3 tell 192.168.200.1, length 28
16:12:49.131784 IP 172.16.30.26.45506 > 172.16.30.28.8472: OTV, flags [I] (0x08), overlay 0, instance 42
ARP, Request who-has 192.168.200.3 tell 192.168.200.1, length 28
16:12:50.155675 IP 172.16.30.26.45506 > 172.16.30.28.8472: OTV, flags [I] (0x08), overlay 0, instance 42
ARP, Request who-has 192.168.200.3 tell 192.168.200.1, length 28
16:12:52.203901 IP 172.16.30.26.45506 > 172.16.30.28.8472: OTV, flags [I] (0x08), overlay 0, instance 42
ARP, Request who-has 192.168.200.3 tell 192.168.200.1, length 28
16:12:53.228516 IP 172.16.30.26.45506 > 172.16.30.28.8472: OTV, flags [I] (0x08), overlay 0, instance 42
ARP, Request who-has 192.168.200.3 tell 192.168.200.1, length 28
16:12:54.251235 IP 172.16.30.26.45506 > 172.16.30.28.8472: OTV, flags [I] (0x08), overlay 0, instance 42
ARP, Request who-has 192.168.200.3 tell 192.168.200.1, length 28
16:12:56.299492 IP 172.16.30.26.45506 > 172.16.30.28.8472: OTV, flags [I] (0x08), overlay 0, instance 42
ARP, Request who-has 192.168.200.3 tell 192.168.200.1, length 28
16:12:57.324093 IP 172.16.30.26.45506 > 172.16.30.28.8472: OTV, flags [I] (0x08), overlay 0, instance 42
ARP, Request who-has 192.168.200.3 tell 192.168.200.1, length 28
16:12:58.347268 IP 172.16.30.26.45506 > 172.16.30.28.8472: OTV, flags [I] (0x08), overlay 0, instance 42
ARP, Request who-has 192.168.200.3 tell 192.168.200.1, length 28
^C
9 packets captured
9 packets received by filter
0 packets dropped by kernel
After some poring over configs I figured out that all the vxlan0 interfaces are being assigned the same MAC address which is what was causing the problem. Changing the MAC addresses of the vxlan0 manually using the below command interfaces made the ping work.
sudo ip link set dev wlan1 address 92:f9:2d:1a:59:6f
sudo ip link set dev wlan1 address 92:f9:2d:1a:59:6d
My question is why does the VxLAN creation create the same MAC address for all 3 machines and how do I make sure each vxlan0 interface has a unique MAC address.
