Varnish ships with a built-in VCL that is deliberately conservative: any request with a Cookie or Authorization header goes straight to the backend, and any response with Set-Cookie is not cached. On a real site, where every browser sends analytics cookies, that means the cache does almost nothing. This guide covers the VCL changes that actually move the hit rate, how to keep serving during backend outages, how to invalidate content safely, how to size storage, and how to prove each change with varnishstat and varnishlog.

Every VCL snippet here was compiled and exercised against the official Docker images for Varnish Cache 7.7.3, 8.0.2 and 9.1.0 with identical results. It uses vcl 4.1 syntax throughout.

Start by measuring the hit rate and why requests miss

You cannot tune what you have not measured. Two commands give you the picture.

varnishstat: the counters

bash
varnishstat -1 -f MAIN.client_req -f MAIN.cache_hit -f MAIN.cache_miss \
  -f MAIN.cache_hitpass -f MAIN.cache_hitmiss -f MAIN.n_lru_nuked \
  -f MAIN.n_object -f SMA.s0.g_bytes -f SMA.s0.g_space -f SMA.Transient.g_bytes
text
MAIN.client_req             13         0.81 Good client requests received
MAIN.cache_hit               5         0.31 Cache hits
MAIN.cache_hitpass           0         0.00 Cache hits for pass.
MAIN.cache_hitmiss           0         0.00 Cache hits for miss.
MAIN.cache_miss              5         0.31 Cache misses
MAIN.n_object                3          .   object structs made
MAIN.n_lru_nuked             0         0.00 Number of LRU nuked objects
SMA.s0.g_bytes             757          .   Bytes outstanding
SMA.s0.g_space       104856843          .   Bytes available
SMA.Transient.g_bytes        0          .   Bytes outstanding

How to read them:

  • Hit ratio is cache_hit / (cache_hit + cache_miss). Requests that were passed never count as either, so also compare cache_hit with client_req: if most requests are neither hit nor miss, VCL is passing them, and that is the problem to fix first.
  • cache_hitpass counts lookups that found a “hit-for-pass” marker, meaning an earlier response was declared uncacheable (usually because of Set-Cookie or Cache-Control: private).
  • cache_hitmiss is the same for “hit-for-miss” markers. A high number here usually means a backend that intermittently sends uncacheable responses for URLs that should be cacheable.
  • n_lru_nuked climbing means storage is full and Varnish is evicting objects to make room. Your cache is too small for the working set (see the storage section).
  • SMA.s0.* is the first malloc storage; SMA.Transient.* is transient storage, which holds short-lived and uncacheable objects.

varnishstat -1 -j emits the same counters as JSON for scripts, and varnishstat with no arguments is a live, sortable view.

varnishncsa and varnishlog: why each request went where it did

The fastest diagnostic is to group requests by how Varnish handled them:

bash
varnishncsa -d -F '%{Varnish:handling}x %U%q' | sort | uniq -c | sort -rn | head
text
      2 hit /page
      2 miss /page
      1 pass /page
      1 miss /list?b=2&a=1&utm_source=x
      1 miss /list?a=1&b=2

-d reads the existing shared memory log instead of waiting for new traffic. A pass on a page that should be cacheable means a cookie or authorization header got through. Two miss lines for what is logically the same URL with different query strings means the cache key is fragmented.

varnishlog shows the full transaction, including every change your VCL made to the URL. To see only the requests that missed:

bash
varnishlog -g request -q 'VCL_call eq "MISS"' -i ReqURL

Other useful queries: -q 'RespStatus >= 500' for errors, -q 'VCL_call eq "PASS"' -i ReqURL -i ReqHeader to see the URL and headers of every passed request (the Cookie or Authorization header that caused it is usually right there), and -q 'Timestamp:Resp[2] > 1.0' for requests that took more than a second. varnishtop -i ReqURL shows the most requested URLs live, which is where to focus first.

varnishhist: latency at a glance

varnishhist draws a live histogram of response times, with hits drawn as | and misses as #. A healthy cache shows a tall column of | far to the left (microseconds) and a smaller cluster of # further right (backend latency). If the # cluster is large, your hit rate is low; if it is far to the right, the backend is slow and grace (below) matters even more. Use -P Beresptime to plot backend response times instead.

Fix the cache key: query strings and cookies

The full VCL used throughout is at the end of this section. Here is what each part does.

Normalize query strings

Marketing parameters and parameter order create a separate cache object for every variation of the same page. Strip the tracking parameters your backend ignores, then sort the rest:

vcl
import std;

sub vcl_recv {
    set req.url = regsuball(req.url, "([?&])(utm_[a-z]+|gclid|fbclid|mc_eid)=[^&]*", "\1");
    set req.url = regsuball(req.url, "&{2,}", "&");
    set req.url = regsub(req.url, "[?&]+$", "");
    set req.url = std.querysort(req.url);
}

With this in place, /list?b=2&a=1&utm_source=x and /list?a=1&b=2 hash to the same object. Only strip parameters you have confirmed the backend does not use; stripping a real parameter serves the wrong content.

Keep only the cookies the backend reads

Since Varnish 6.4, vmod_cookie ships with Varnish itself. Use it to keep an allow-list of cookies and drop everything else (analytics, A/B testing, chat widgets):

vcl
import cookie;

sub vcl_recv {
    if (req.url ~ "^/(assets|static)/" || req.url ~ "\.(css|js|png|jpg|svg|woff2)(\?|$)") {
        unset req.http.cookie;
        return (hash);
    }
    if (req.http.cookie) {
        cookie.parse(req.http.cookie);
        cookie.keep("sessionid,csrftoken");
        set req.http.cookie = cookie.get_string();
        if (req.http.cookie == "") {
            unset req.http.cookie;
        }
    }
}

After this, a request with only _ga and _gid cookies is a cache hit, while a logged-in user with sessionid still falls through to the built-in VCL and is passed. That is the correct default: never cache personalized responses unless the backend explicitly varies on something safe.

Do not unset beresp.http.Set-Cookie globally to “make things cacheable”. If the backend sets a session cookie on a page and you cache that response, every visitor receives the same session.

When to touch vcl_hash

The default hash is the URL plus the Host header (or the server IP when there is no Host). Add to it only for real variation that the backend does not express with Vary, for example a language cookie you kept:

vcl
sub vcl_hash {
    if (req.http.cookie ~ "lang=") {
        hash_data(regsub(req.http.cookie, "^.*lang=([a-z]{2}).*$", "\1"));
    }
}

Without a return, the built-in vcl_hash still runs afterward and adds URL and host. Prefer a correct Vary header from the backend where you can, but keep Vary values few: Vary: User-Agent creates one object per browser version and effectively disables the cache.

Grace and keep: serve stale instead of errors

Every object has three timers:

  • ttl: how long the object is fresh. Set from Cache-Control: s-maxage or max-age, then Expires, then the default_ttl parameter (120 seconds by default).
  • grace: how long after the TTL an expired object may still be delivered while Varnish refreshes it in the background.
  • keep: how long after TTL plus grace the object is kept only so Varnish can send a conditional request (If-Modified-Since or If-None-Match) and get a cheap 304 instead of a full body.
vcl
sub vcl_recv {
    # While the backend is healthy, accept at most 10s of staleness.
    if (std.healthy(req.backend_hint)) {
        set req.grace = 10s;
    }
}

sub vcl_backend_response {
    set beresp.grace = 6h;
    set beresp.keep = 1d;

    if (beresp.status >= 500) {
        if (bereq.is_bgfetch) {
            return (abandon);
        }
        set beresp.uncacheable = true;
        set beresp.ttl = 10s;
        return (deliver);
    }
}

This is the pattern from the Varnish grace documentation, with one addition. Normally a client gets at most 10 seconds of stale content while a background fetch refreshes it. When the health probe marks the backend sick, req.grace is left alone, so the full six hours of beresp.grace applies and visitors keep getting pages instead of errors. return (abandon) on a failed background fetch keeps the existing stale object rather than replacing it with a 503.

Grace only works with a health probe on the backend, so std.healthy() has something to report:

vcl
backend default {
    .host = "app";
    .port = "8080";
    .connect_timeout = 2s;
    .first_byte_timeout = 30s;
    .probe = {
        .url = "/healthz";
        .interval = 2s;
        .timeout = 1s;
        .window = 5;
        .threshold = 3;
    }
}

In testing, a page with Cache-Control: max-age=5 was fetched once, the backend was taken down, and 12 seconds later the request was still answered from cache (Age: 12, X-Cache: HIT) while varnishadm backend.list showed the backend as sick.

Bans vs purges

Purge removes one object, with all its Vary variants, immediately. It is the right tool when you know the exact URL that changed:

vcl
acl purgers {
    "127.0.0.1";
    "172.16.0.0"/12;
}

sub vcl_recv {
    if (req.method == "PURGE") {
        if (client.ip !~ purgers) {
            return (synth(405, "Not allowed"));
        }
        return (purge);
    }
}
bash
curl -X PURGE http://varnish.internal/products/42

Ban invalidates everything matching an expression, for example a whole section of the site. Bans are not applied immediately: they go on a ban list and each cached object is checked against newer bans when it is next looked up, or by the background “ban lurker” thread. Write bans against obj.* fields only, so the lurker can evaluate them without a request. That is why the VCL copies the URL and host onto the object:

vcl
sub vcl_recv {
    if (req.method == "BAN") {
        if (client.ip !~ purgers) {
            return (synth(405, "Not allowed"));
        }
        if (!req.http.x-ban-url) {
            return (synth(400, "Missing x-ban-url"));
        }
        ban("obj.http.x-host == " + req.http.host + " && obj.http.x-url ~ " + req.http.x-ban-url);
        return (synth(200, "Ban added"));
    }
}

sub vcl_backend_response {
    set beresp.http.x-url = bereq.url;
    set beresp.http.x-host = bereq.http.host;
}

sub vcl_deliver {
    unset resp.http.x-url;
    unset resp.http.x-host;
}
bash
curl -X BAN -H 'x-ban-url: ^/products/' http://varnish.internal/
varnishadm ban.list

A ban that references req.* (for example req.url ~ ...) cannot be processed by the lurker, so it waits until every old object is requested. On a large cache the ban list grows without bound and every lookup gets slower. Watch MAIN.bans in varnishstat: if it keeps growing, some bans are not lurker friendly.

You can also ban from the CLI without any VCL: varnishadm ban 'obj.http.x-url ~ ^/products/'.

Storage sizing

Varnish keeps objects in the storage backends you pass with -s:

bash
varnishd -a :80 -f /etc/varnish/default.vcl \
  -s malloc,4G \
  -s Transient=malloc,512m
  • Size malloc from the working set, not the RAM. Look at n_lru_nuked: if it is zero and SMA.s0.g_space stays large, the cache is big enough. If objects are being nuked, either add memory or shorten TTLs on content that is rarely reused.
  • Budget memory beyond -s. The storage size caps the net allocation, but the process also needs on the order of 1 KB per object outside storage, and allocator fragmentation can push real usage well above the configured size. On a machine dedicated to Varnish, giving storage roughly three quarters of RAM leaves room for that; on hosts with millions of small objects, leave more.
  • Bound Transient. Transient storage, used for short-TTL and uncacheable objects such as hit-for-pass markers and passed response bodies in flight, is unlimited by default. A traffic spike of large uncacheable responses can make varnishd run out of memory. Give it an explicit limit and watch SMA.Transient.g_bytes.
  • file storage puts objects in a memory-mapped file. It helps only when the working set is far larger than RAM and the disk is fast; pre-create the file (for example with dd) so it is not fragmented.

On the official Docker image, the entrypoint already adds -s malloc,$VARNISH_SIZE (100M by default), so set the VARNISH_SIZE environment variable rather than passing a second unnamed -s malloc argument, which would create an additional storage instead of resizing the first.

The complete VCL

vcldefault.vcl
vcl 4.1;

import std;
import cookie;

backend default {
    .host = "app";
    .port = "8080";
    .connect_timeout = 2s;
    .first_byte_timeout = 30s;
    .probe = {
        .url = "/healthz";
        .interval = 2s;
        .timeout = 1s;
        .window = 5;
        .threshold = 3;
    }
}

acl purgers {
    "127.0.0.1";
    "172.16.0.0"/12;
}

sub vcl_recv {
    if (req.method == "PURGE") {
        if (client.ip !~ purgers) {
            return (synth(405, "Not allowed"));
        }
        return (purge);
    }
    if (req.method == "BAN") {
        if (client.ip !~ purgers) {
            return (synth(405, "Not allowed"));
        }
        if (!req.http.x-ban-url) {
            return (synth(400, "Missing x-ban-url"));
        }
        ban("obj.http.x-host == " + req.http.host + " && obj.http.x-url ~ " + req.http.x-ban-url);
        return (synth(200, "Ban added"));
    }

    set req.url = regsuball(req.url, "([?&])(utm_[a-z]+|gclid|fbclid|mc_eid)=[^&]*", "\1");
    set req.url = regsuball(req.url, "&{2,}", "&");
    set req.url = regsub(req.url, "[?&]+$", "");
    set req.url = std.querysort(req.url);

    if (req.url ~ "^/(assets|static)/" || req.url ~ "\.(css|js|png|jpg|svg|woff2)(\?|$)") {
        unset req.http.cookie;
        return (hash);
    }

    if (req.http.cookie) {
        cookie.parse(req.http.cookie);
        cookie.keep("sessionid,csrftoken");
        set req.http.cookie = cookie.get_string();
        if (req.http.cookie == "") {
            unset req.http.cookie;
        }
    }

    if (std.healthy(req.backend_hint)) {
        set req.grace = 10s;
    }
}

sub vcl_backend_response {
    set beresp.http.x-url = bereq.url;
    set beresp.http.x-host = bereq.http.host;

    set beresp.grace = 6h;
    set beresp.keep = 1d;

    if (beresp.status >= 500) {
        if (bereq.is_bgfetch) {
            return (abandon);
        }
        set beresp.uncacheable = true;
        set beresp.ttl = 10s;
        return (deliver);
    }
}

sub vcl_deliver {
    unset resp.http.x-url;
    unset resp.http.x-host;
    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
    } else {
        set resp.http.X-Cache = "MISS";
    }
}

Validate before loading: varnishd -C -f /etc/varnish/default.vcl > /dev/null compiles the VCL and prints errors without starting anything. Note that the backend .host must resolve at compile time. Load it into a running instance without a restart (and without losing the cache) with:

bash
varnishadm vcl.load v2 /etc/varnish/default.vcl
varnishadm vcl.use v2

FAQ

Why is my hit rate low even though pages have Cache-Control: public?

Because the built-in VCL passes any request still carrying a Cookie header at the end of your vcl_recv, before it ever looks at the response headers. Run varnishlog -g request -q 'VCL_call eq "PASS"' -i ReqURL -i ReqHeader and look at which cookies the passed requests carried; add the ones the backend does not need to your strip logic, or keep only the ones it does with cookie.keep().

Should I return (hash) early in vcl_recv?

Only for requests you are sure are safe to cache regardless of headers, such as static assets after removing cookies. An early return (hash) skips the built-in vcl_recv, which is what passes Authorization requests and non-GET methods, so returning early for everything can cache private or POST responses.

How do I test VCL changes against production hostnames before switching DNS?

Point curl at the Varnish node while keeping the real hostname, with curl --resolve www.example.com:80:10.0.0.5 -I http://www.example.com/, and watch the X-Cache and Age headers. See cURL Host Header: Set It With -H, --resolve or --connect-to.