Rewrite Vary header before stored in proxy_cache

[Maxim Dounin]

Hello!

On Mon, Oct 24, 2016 at 06:38:25AM +0200, Lucas Rolff wrote:

> Hi guys,
> 
> I'm building a small nginx reverse proxy to take care of a bunch of static
> files for my clients - and it works great.
> 
> One thing I'm facing though is that some client sites sent "Vary:
> Accept-Encoding, User-Agent" - which gives an awful cache hit rate - since
> proxy_cache takes this into account, unless I use something like
> "proxy_ignore_headers Vary;"
> 
> But ignoring Vary headers can cause other issues such as gzipped content
> being sent to a non-gzip client.
> 
> So I'm looking for a way to basically rewrite the vary header to "Vary:
> Accept-Encoding" before storing it in proxy_cache - but I wonder if this is
> even possible in nginx, and if yes - can you give any pointers?
> 
> I found a temporary fix, and that is to ignore the Vary header, and using a
> custom variable as a part of the cache key, that is either "", "gzip" or
> "deflate" (I use a map to look at the Accept-Encoding header from the
> client).
> 
> This works great - but I rather keep the cache key a bit clean (since I'll
> use it later)
> 
> Do you guys have any recommendations how to make this happen?

The best possible solution I can think of is to ask the client to fix 
the Vary header it returns.  Using User-Agent in Vary is something 
one shouldn't use without a very good reason, and if there a 
reason - it's likely a bad idea to strip from the Vary header.
And if there are no reasons, then it shouldn't be returned in the 
first place.

> Also as a side note, if I remove the custom variable from the cache key,
> how would one actually purge the file then? I assume I have to send
> different purge requests, since the cached file is based on the Vary:
> accept-encoding - so I'd have to purge at least the amount of cached
> encodings right?

When using purge as availalbe in nginx-plus 
(http://nginx.org/r/proxy_cache_purge), it takes care of removing 
all cached variants, much like it does for wildcard purge requests.

> Also I could opt for another way, and that's always requesting a
> uncompressed file from the origin (Is it simply not sending the
> accept-encoding header, or should I do something else?), and then on every
> request either decide to gzip it or not - the downside I see here, is the
> fact that most clients request gzip,deflate content, so having to compress
> on every request will use additional CPU resources.

This can be done easily, just

    proxy_set_header Accept-Encoding "";

should be enough.  Alternatively, you can use

    proxy_set_header Accept-Encoding gzip;
    gunzip on;

to always ask gzipped resources and gunzip them when needed, see 
http://nginx.org/en/docs/http/ngx_http_gunzip_module.html.

-- 
Maxim Dounin
http://nginx.org/