Set a TTL
As stated in the built-in VCL: Varnish treats the no-cache directive in the same way as private & no-store: the content will not end up in the cache.
For ETag-based revalidation that poses a problem, because there is nothing to compare.
My advice would be to set a low TTL to ensure it ends up in the cache. I would recommend using the following Cache-Control header:
Cache-Control: public, s-maxage=3, must-revalidate
Make Varnish understand must-revalidate
Another issue is that Varnish doesn't understand the must-revalidate directive, whereas it does support the stale-while-revalidate directive.
Adding must-revalidate has the same effect as stale-while-revalidate=0: we cannot serve stale content when the object expires and require immediate synchronous validation.
This sets the internal grace timer to zero.
However, with the following VCL code, you can make Varnish respect the must-revalidate directive:
sub vcl_backend_response {
if(beresp.http.Cache-Control ~ "must-revalidate") {
set beresp.grace = 0s;
}
}
Increase the keep time
By setting the grace value to zero and assigning a low TTL, the objects will expire fairly quickly and won't be around long enough for revalidation.
As explained in https://stackoverflow.com/questions/68177623/varnish-default-grace-behavior/68207764#68207764 Varnish has a set of timers it uses
to decide on expiration, revalidation & tolerated staleness.
My suggestion would be to increase the keep timer in VCL. This timer ensures that expired and out-of-grace objects are kept around, without running the risk of serving stale content.
The only reason why the keep timer exists is for ETag revalidation, so that's exactly what you need.
I suggest using the following VCL to support must-revalidate & increase the keep timer:
sub vcl_backend_response {
set beresp.keep = 1d;
if(beresp.http.Cache-Control ~ "must-revalidate") {
set beresp.grace = 0s;
}
}
This snippet increases the keep timer to a day, allowing expired content to stay in the cache for a day while revalidation takes place.
Watch out with cookies (& authorization headers)
Despite having all the bits and pieces in place to support ETag revalidation and store objects in the cache, it's important that relevant requests don't bypass the cache. That has nothing to do with Cache-Control headers, but with request headers.
If you take a look at the built-in VCL for the vcl_recv subroutine, you'll notice that by default Varnish bypasses the cache for requests containing an Authorization header or a Cookie header.
If your website uses cookies, please read the following tutorial to learn how to remove tracking cookies that could mess up your hit rate: https://www.varnish-software.com/developers/tutorials/removing-cookies-varnish/
Varnish test case as a proof of concept
Store the following VTC content in etag.vtc and run varnishtest etag.vtc to validate the test case:
varnishtest "Testing ETag & 304 status codes in Varnish"
server s1 {
# First backend request is not a conditional one
# Return 200 with the ETag
rxreq
expect req.method == "GET"
expect req.http.If-None-Match == <undef>
txresp -status 200
-hdr "Cache-Control: public, s-maxage=3, must-revalidate"
-hdr "ETag: abc123" -bodylen 7
# Second backend request is a conditional one with a matching ETag
# Return a 304
rxreq
expect req.method == "GET"
expect req.http.If-None-Match == "abc123"
txresp -status 304 \
-hdr "Cache-Control: public, s-maxage=3, must-revalidate" \
-hdr "ETag: abc123"
# Third backend request is a conditional where the Etag doesn't match
# The content has been updated
# Return a 200 with the updated content and the new Etag
rxreq
expect req.method == "GET"
expect req.http.If-None-Match == "abc123"
txresp -status 200 \
-hdr "Cache-Control: public, s-maxage=3, must-revalidate" \
-hdr "ETag: xyz456" -bodylen 8
} -start
varnish v1 -vcl+backend {
sub vcl_backend_response {
set beresp.keep = 1d;
if(beresp.http.Cache-Control ~ "must-revalidate") {
set beresp.grace = 0s;
}
}
} -start
client c1 {
# Send a regular request
# Expect a cache miss
# Trigger the first backend response
# Return 200 with a response body
txreq -url "/"
rxresp
expect resp.status == 200
expect resp.http.Age == "0"
expect resp.http.ETag == "abc123"
expect resp.http.Cache-Control == "public, s-maxage=3, must-revalidate"
expect resp.bodylen == 7
# Wait for 1 second
delay 1
# Send a conditional request with an If-None-Match header
# Should be a cache hit
# Should return a 304 without a response body
# Varnish is responsible for this 304
txreq -url "/" -hdr "If-None-Match: abc123"
rxresp
expect resp.status == 304
expect resp.http.Age != "0"
expect resp.http.ETag == "abc123"
expect resp.http.Cache-Control == "public, s-maxage=3, must-revalidate"
expect resp.bodylen == 0
# Wait for 3 seconds
delay 3
# Send a conditional request with an If-None-Match header
# Should be a cache miss
# Trigger the second backend response
# Should return a 304 without a response body
# The server is responsible for this 304
txreq -url "/" -hdr "If-None-Match: abc123"
rxresp
expect resp.status == 304
expect resp.http.Age == "0"
expect resp.http.ETag == "abc123"
expect resp.http.Cache-Control == "public, s-maxage=3, must-revalidate"
expect resp.bodylen == 0
# Wait for 4 seconds
delay 4
# Send a conditional request with an If-None-Match header
# Should be a cache miss
# Trigger the third backend response
# Should return a 200 with a response body
# Content has been updated: a new Etag is returned
txreq -url "/" -hdr "If-None-Match: abc123"
rxresp
expect resp.status == 200
expect resp.http.Age == "0"
expect resp.http.ETag == "xyz456"
expect resp.http.Cache-Control == "public, s-maxage=3, must-revalidate"
expect resp.bodylen == 8
} -run
Conclusion
If you follow the steps I described, your Varnish setup will support ETag revalidation in 2 ways:
- From the client to the cached object in Varnish while the object is still fresh
- From Varnish to the origin server when the object has expired, but while it's still kept around in the cache due to the
keep timer
Please also consider increasing the TTL of the cache if your content allows it. This all depends on how much pages/resources change.
Please also realize that keeping objects around longer (due to an increased keep value), will fill up the cache. When the cache is full an LRU strategy is applied to remove the least popular object from the cache.
If set beresp.keep = 1d; is too much and Varnish starts removing objects from the cache because it's full, consider lowering the keep value.
