How can I get nginx not to override x-forwarded-for when proxying?

Question

I have an nginx server behind a load balancer, the nginx server passes requests on to a variety of services, but in this case a docker container running apache. The load balancer sets an X-Forwarded-For correctly, but by the time it gets to the docker container, X-Forwarded-For has been set to the LB IP.

I have this in nginx config:

/etc/nginx/conf.d/real_ip.conf
set_real_ip_from {{LB IP}};
real_ip_header X-Real-IP;
real_ip_recursive on;

and this is the virtualhost:

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name *.domain domain;
    include /etc/nginx/snippets/domain_ssl.conf;
add_header X-Nginx-Debug "hi";
proxy_pass_request_headers on;
location    / {
    proxy_pass_request_headers on;
    proxy_pass  http://container-php;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Remote-Addr $remote_addr;
    proxy_set_header X-Real-IP $http_x_real_ip;
    proxy_set_header X-Header-Test "Hello World - $http_x_forwarded_for";
    proxy_set_header X-Forwarded-Proto $scheme;
  }
}

But what I get from the container is:

array(19) {
  ["Connection"]=>
  string(7) "upgrade"
  ["Host"]=>
  string(19) "domain"
  ["X-Forwarded-For"]=>
  string(12) "{{LB IP}}"
  ["X-Header-Test"]=>
  string(13) "Hello World -"
  ["X-Forwarded-Proto"]=>
  string(5) "https"
  ["cache-control"]=>
  string(9) "max-age=0"
  ["sec-ch-ua"]=>
  string(64) "" Not;A Brand";v="99", "Google Chrome";v="97", "Chromium";v="97""
  ["sec-ch-ua-mobile"]=>
  string(2) "?0"
  ["sec-ch-ua-platform"]=>
  string(9) ""Windows""
  ["upgrade-insecure-requests"]=>
  string(1) "1"
  ["user-agent"]=>
  string(114) "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36"
  ["accept"]=>
  string(135) "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
  ["sec-fetch-site"]=>
  string(4) "none"
  ["sec-fetch-mode"]=>
  string(8) "navigate"
  ["sec-fetch-user"]=>
  string(2) "?1"
  ["sec-fetch-dest"]=>
  string(8) "document"
  ["accept-encoding"]=>
  string(17) "gzip, deflate, br"
  ["accept-language"]=>
  string(26) "en-GB,en-US;q=0.9,en;q=0.8"
}

Notably X-Real-IP, X-Fowarded-For don't seem to be set, nor does remote_addr. Files served directly from nginx have x-forwarded-for set properly, so the LB is sending down the right header.

Have I missed a step?

score 5 · Answer 1 · answered Feb 01 '22 at 09:40

I think the problem is in your real ip configuration.

set_real_ip_from {{LB IP}};

real_ip_header X-Real-IP;

real_ip_recursive on;

When real_ip_header should be (in your case) set to X-Forwarded-For. I'll assume your X-Forwarded-For header from LB looks like this :

X-Forwarded-For: {{Original client ip}}, {{LB ip}}

So when you set real_ip_header (The header used to replace client ip) to X-Forwarded-For it will match the original client ip. The Original client should now be under the variable $realip_remote_addr, which you can address to proxy_set_header X-Forwarded-For :

proxy_set_header X-Forwarded-For $realip_remote_addr

Please let me know if I have been of any help !

score 2 · Answer 2 · answered Jan 21 '22 at 17:12

This is the statement overriding the X-Forwarded-For header:

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

Assuming you want to preserve the original client IP in that header, you should write:

proxy_set_header X-Forwarded-For $remote_addr;

score 1 · Answer 3 · answered Nov 20 '24 at 13:13

In case anyone would encounter this issue. Here is what I learned after many hours of searching.

onlyoffice running inside docker will nowerdays redirect anything that goes to port 80 (http) to port 443 (https)
onlyoffice running inside docker will redirect path / to /welcome, and so it is for docservice . The variable it uses is $the_host, which is mapped from $http_x_forwarded_host, and this $http_x_forwarded_host stands for X-Forwarded-Host. You can see by yourself from config under /etc/nginx/includes/xxx.conf.

map $http_x_forwarded_host $the_host {
    default $http_x_forwarded_host;
    "" $this_host;
}

Therefore, if you have another reverse proxy for onlyoffice. You must pass the right "X-Forwarded-Host" value to the nginx inside the container.
For example, in Nginx Proxy Manager, you need to add a custom location and add this into advanced edit box.

proxy_set_header X-Forwarded-Host $http_host;

One more thing about $http_host, this param includes the requested port in case you are using other https port than the default 443. If you are using Apache, you probably won't have any issue using default config. Because Apache uses $http_host by default. While Nginx only uses $host by default which does not include the port. So if not explicitly anounced, nginx will not pass the port to backend reverse proxy or anything in the chain behind.

score 0 · Answer 4 · answered Jan 26 '22 at 18:16

From your description your problem isn't directly related to nginx but with apache.

Depending on your configuration your traffic flow might look something like:

outside -> nginx -> apache -> php (running as fpm)

or

outside -> nginx -> apache + php module

You should look into the apache config to make sure that out if apache isn't discarding the x-forwarded-for and x-real-ip headers when passing the request to PHP.

If your traffic flow is in line with the first example, you have apache proxying the request also to php which could lead to a lot more issues if nginx and apache aren't "in sync".

If you were handling the php proxying with nginx alone you would just need to add the following to your php location config:

location ~ \.php$ {
        #...other rules
        fastcgi_param REMOTE_ADDR $http_x_real_ip;
        #...other rules
}

This way, PHP's REMOTE_ADDR would be correctly set.

In regards to your nginx config if your nginx {{LB IP}} is static, you can set it directly in the config.

/etc/nginx/conf.d/real_ip.conf
set_real_ip_from {{LB IP}};
real_ip_recursive on;

You don't need real_ip_header X-Real-IP; in your config file. That will override the set_real_ip_from directive.

Keep in mind that you need to have the real_ip module enabled in nginx in order for this to work.

score 0 · Accepted Answer · answered Feb 01 '22 at 15:27

I eventually solved this in a way that works, but doesn't actually solve the root problem. As far as I can tell, the upstream LB is setting "X-Forwarded-For" to the remote address, and nginx's magic attempting set that correctly for the reverse proxy was always getting it wrong and setting it to the LB address or an empty string.

Instead, I switched from HTTP to PROXY protocol for the LB->Server bit, set the following in /etc/nginx/proxy_params:

proxy_set_header X-Real-IP       $proxy_protocol_addr;
proxy_set_header X-Forwarded-For $proxy_protocol_addr;

and this in conf.d:

 /etc/nginx/conf.d/real_ip.conf
set_real_ip_from {{ LB_IP }};
real_ip_header proxy_protocol;

adapted from this:

https://www.x33u.org/docs/kubernetes-stuff/hetzner-loadbalancer-setup/

and now everything works.

How can I get nginx not to override x-forwarded-for when proxying?

5 Answers5