How to set robots.txt globally in nginx for all virtual hosts

Question

I am trying to set robots.txt for all virtual hosts under nginx http server. I was able to do it in Apache by putting the following in main httpd.conf:

<Location "/robots.txt">
    SetHandler None
</Location>
Alias /robots.txt /var/www/html/robots.txt

I tried doing something similar with nginx by adding the lines given below (a) within nginx.conf and (b) as include conf.d/robots.conf

location ^~ /robots.txt {
        alias /var/www/html/robots.txt;
}

I have tried with '=' and even put it in one of the virtual host to test it. Nothing seemed to work.

What am I missing here? Is there another way to achieve this?

score 103 · Answer 1 · edited Jul 28 '20 at 11:39

103

You can set the contents of the robots.txt file directly in the nginx config:

location = /robots.txt { return 200 "User-agent: *\nDisallow: /\n"; }

It is also possible to add the correct Content-Type:

location = /robots.txt {
   add_header Content-Type text/plain;
   return 200 "User-agent: *\nDisallow: /\n";
}

edited Jul 28 '20 at 11:39

Vishal Singh

103

answered Jan 26 '16 at 14:07

Konstantin V Bekreyev

1,131

score 13 · Answer 2 · answered Oct 29 '13 at 11:01

Are there other rules that are defined? Maybe common.conf or another conf file in included which is over-riding your config. One of the following should definitely work.

location /robots.txt { alias /home/www/html/robots.txt; }
location /robots.txt { root /home/www/html/;  }

Nginx runs all "regexp" locations in order of their appearance. If any "regexp" location succeeds, Nginx will use this first match. If no "regexp" location succeeded, Nginx uses the ordinary location found on the previous step.
"regexp" locations have precedence over "prefix" locations

score 7 · Accepted Answer · answered Oct 29 '13 at 12:36

location cannot be used inside http block. nginx does not have global aliases (i.e., aliases that can be defined for all vhosts). Save your global definations in a folder and include those.

server {
  listen 80;
  root /var/www/html;
  include /etc/nginx/global.d/*.conf;
}

score 1 · Answer 4 · edited Apr 05 '24 at 20:11

1

You could also just serve it directly:

location /robots.txt {
   return 200 "User-agent: *\nDisallow: /\n";
}

edited Apr 05 '24 at 20:11

eduherminio

103

answered Aug 26 '20 at 13:54

Ben Bieler

119

score -1 · Answer 5 · answered Jul 04 '18 at 21:54

I had the same issue with the acme challanges, but the same principle applies to your case as well.

What I did to solve this issue was to move all my sites to a non-standard port, I picked 8081, and created a virtual server listening on port 80. It proxies all requests to 127.0.0.1:8081, except the ones to .well-known. This acts almost as a global alias, with one extra hop, but that shouldn't cause a significant drop in performance due to the async nature of nginx.

upstream nonacme {
  server 127.0.0.1:8081;
}

server {
  listen 80;

  access_log  /var/log/nginx/acme-access.log;
  error_log   /var/log/nginx/acme-error.log;

  location /.well-known {
    root /var/www/acme;
  }

  location / {
    proxy_set_header    Host                $http_host;
    proxy_set_header    X-Real-IP           $remote_addr;
    proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
    proxy_set_header    X-Forwarded-Proto   $scheme;
    proxy_set_header    X-Frame-Options     SAMEORIGIN;

    # WebSocket support (nginx 1.4)
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";

    proxy_pass http://nonacme;
  }
}

How to set robots.txt globally in nginx for all virtual hosts

5 Answers5