I have a setup where I want a transparent TCP proxy in front of some HTTPS services. The services themselves deal with certificates so I'm trying to avoid a HTTP proxy here. I also need the services to get the client's original IP.
The setup is like this:
- 192.168.86.33: HAProxy
- 192.168.86.31: Service1
- 192.168.86.36: Service2 All three run Raspberry Pi OS.
Based on docs it seems like HAProxy is a good tool for the job but I can't get it to work. It works until I add source 0.0.0.0 usesrc clientip but the service sees the IP of the proxy instead of the client.
When I add source 0.0.0.0 usesrc clientip I get this in the client: Error: Client network socket disconnected before secure TLS connection was established
And this in the HAProxy log when run with strace:
[{EPOLLIN, {u32=15, u64=15}}], 200, 60000, NULL, 8) = 1
clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=304312352}) = 0
read(15, "c", 1024) = 1
read(15, 0x7fe9410d50, 1024) = -1 EAGAIN (Resource temporarily unavailable)
setsockopt(17, SOL_TCP, TCP_NODELAY, [1], 4) = 0
recvfrom(17, 0x55a7d68c80, 15360, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 18
fcntl(18, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
setsockopt(18, SOL_TCP, TCP_NODELAY, [1], 4) = 0
setsockopt(18, SOL_IP, IP_BIND_ADDRESS_NO_PORT, [1], 4) = 0
setsockopt(18, SOL_IP, IP_TRANSPARENT, [1], 4) = 0
setsockopt(18, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
bind(18, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.86.161")}, 16) = 0
connect(18, {sa_family=AF_INET, sin_port=htons(30989), sin_addr=inet_addr("192.168.86.36")}, 16) = -1 EINPROGRESS (Operation now in progress)
epoll_ctl(4, EPOLL_CTL_ADD, 17, {EPOLLIN|EPOLLRDHUP, {u32=17, u64=17}}) = 0
epoll_ctl(4, EPOLL_CTL_ADD, 18, {EPOLLIN|EPOLLOUT|EPOLLRDHUP, {u32=18, u64=18}}) = 0
clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=307519852}) = 0
epoll_pwait(4, [{EPOLLIN, {u32=17, u64=17}}], 200, 5001, NULL, 8) = 1
clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=307766259}) = 0
recvfrom(17, "\26\3\1\2\0\1\0\1\374\3\3\277\357D\241\236\245Gw\361\16\6\273\234\212D\245\305\241.\10o"..., 15360, 0, NULL, NULL) = 517
My full haproxy.cfg below. I have temporarily removed out one of the backends:
defaults
mode tcp
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
frontend db
bind 192.168.86.33:30989 transparent
default_backend databases
backend databases
source 0.0.0.0 usesrc clientip
server db2 192.168.86.36:30989
Connecting directly to 192.168.86.36 works. Connecting via the proxy works if I remove the "source" line but my service sees the wrong IP.
I believe I have the right kernel modules loaded:
lsmod | grep proxy
nft_tproxy 16384 0
nf_tproxy_ipv6 16384 1 nft_tproxy
nf_tproxy_ipv4 16384 1 nft_tproxy
nf_defrag_ipv6 20480 3 nf_conntrack,xt_socket,nft_tproxy
nf_defrag_ipv4 16384 3 nf_conntrack,xt_socket,nft_tproxy
nf_tables 237568 212 nft_compat,nft_tproxy,nft_chain_nat,nft_socket
ipv6 557056 50 nf_tproxy_ipv6,bridge,br_netfilter,nf_socket_ipv6
These are the iptables and routes set on the proxy:
#!/bin/bash
iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 111
iptables -t mangle -A DIVERT -j ACCEPT
ip rule add fwmark 111 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100
Based on the response here I have also set the proxy as the default gateway on the service machine:
ip route show
default via 192.168.86.33 dev wlan0 src 192.168.86.36 metric 303
Final things I've tried:
- Recompiled haproxy with USE_LINUX_TPROXY=1:
OPTIONS = USE_LINUX_TPROXY=1 - I have set net.ipv4.ip_forward & net.ipv4.ipnonlocalbind to 1
Any help as to where I am going wrong would be much appreciated!
