11

I have recently run into an issue after switching to Cloudflare, and the solution is to basically stop Cloudflare from caching 404 responses.

In our load-balanced multi-server setup, occasional 404s happen, but they're quickly fixed by rsync (via lsyncd). Before Cloudflare, a re-request to the 404ed file would very quickly become 200 as rsync does its job.

However, since Cloudflare caches all data based on the caching header, and neither apache nor nginx send a no-cache header for 404s, Cloudflare ends up caching the 404 response for a while.

I've been searching for a solution to globally add such a header for 404s in both apache and nginx (globally, for all hosted domains), but so far have come up blank.

Can anyone help?

Thank you.

Artem Russakovskii
  • 1,023

5 Answers5

7

Can't you get by with using an error_page directive, and then handle the location separately with the added header?

e.g. in Nginx:

    server {
      ...
      error_page 404 /404.html;
      location = /404.html {
        root   /usr/share/nginx/html;
        add_header Cache-Control "no-cache" always;
      }
    }
jimmiw
  • 118
7

You can do it this way too :

map $status $cache_header {
    default <for_other_codes>;
    404     "no-cache";
}


server {

    [ ... ]

    add_header "Cache-Control" $cache_header always;

}
Xavier Lucas
  • 13,505
6

In apache 2.4, you could try something like:

FileETag None
<IfModule mod_headers.c>
    Header always set Cache-Control "max-age=0, no-cache, no-store, must-revalidate" "expr=%{REQUEST_STATUS} == 404"
    Header always set Expires "Wed, 11 Jan 1984 05:00:00 GMT" "expr=%{REQUEST_STATUS} == 404"
</IfModule>

The always is important because this is a:

You're adding a header to a locally generated non-success (non-2xx) response, such as a redirect, in which case only the table corresponding to always is used in the ultimate response.

You said all 404s, but for full reference of course it might make sense to wrap that in a <FilesMatch> or <LocationMatch> to limit the scope.

I believe this is a new capability in apache 2.4 as using expr conditionals is not the in the 2.2 version of the mod_headers documentation.

curl -I [foo] test without this config:

HTTP/1.1 404 Not Found
Date: Thu, 24 May 2018 17:44:29 GMT
Server: Apache/2.4.18 (Ubuntu)
Content-Type: text/html; charset=iso-8859-1

curl -I [foo] test with this config:

HTTP/1.1 404 Not Found
Date: Thu, 24 May 2018 17:44:42 GMT
Server: Apache/2.4.18 (Ubuntu)
Cache-Control: max-age=0, no-cache, no-store, must-revalidate
Pragma: no-cache
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Content-Type: text/html; charset=iso-8859-1

Sources:

http://httpd.apache.org/docs/current/mod/mod_headers.html

cayleaf
  • 503
1

We ran into this as well on Cloudflare, and I'm not sure the other Nginx answers solve the issue because AFAIK, Nginx location blocks do not inherit Cache-Control headers from their parents, so adding a map at the top to set headers did not affect Nginx error_page blocks. This is what worked for us:

location ~* \.(?:css|eot|ico|js|jpg|jpeg|gif|png)$ {
  error_page 404 = @not_found; # If 404, redirect to the named location @not_found.

Cache fingerprinted static assets for 30 days.


See https://developers.cloudflare.com/cache/concepts/default-cache-behavior.


add_header Cache-Control 'public,max-age=2592000,must-revalidate,proxy-revalidate';
}


location @not_found {
  add_header Cache-Control 'private,no-cache,no-store' always; # Always set Cache-Control header for 404 responses.
}
Dom Eden
  • 11
-1

my five cents on the issue -

in our PHP project we have few 404 pages, so I decide to do it on PHP level using PHP header() functions

Nick
  • 902