Rewrite Vary header before stored in proxy_cache

[Lucas Rolff]

Hi Maxim,

Thank you a lot for the reply!

> The best possible solution I can think of is to ask the client to fix the Vary header it returns

I completely agree, but sometimes it's hard to ask customers to do this, 
but I do try to do it as often as possible.


> When using purge as availalbe in nginx-plus (http://nginx.org/r/proxy_cache_purge), it takes care of removing all cached variants, much like it does for wildcard purge requests.
Ahh cool! Nice one - maybe we'll be lucky that it gets to the open 
source version one day ;)

> This can be done easily, just
>
>      proxy_set_header Accept-Encoding "";
>
> should be enough.  Alternatively, you can use
>
>      proxy_set_header Accept-Encoding gzip;
>      gunzip on;
>
> to always ask gzipped resources and gunzip them when needed, see
> http://nginx.org/en/docs/http/ngx_http_gunzip_module.html.
This is actually what I ended up doing, and it seems to work perfectly - 
still I have to gunzip if the client doesn't support gzip in first 
place, but the percentage is very minimal these days, so it seems like 
the best option, not only I save a bunch of storage (due to compression 
and only storing the file once and not 3 times) - but also makes purging 
super easy!

Once again, thanks a lot!

-- 
Best Regards,
Lucas Rolff


Maxim Dounin wrote:
> Hello!
>
> On Mon, Oct 24, 2016 at 06:38:25AM +0200, Lucas Rolff wrote:
>
>> Hi guys,
>>
>> I'm building a small nginx reverse proxy to take care of a bunch of static
>> files for my clients - and it works great.
>>
>> One thing I'm facing though is that some client sites sent "Vary:
>> Accept-Encoding, User-Agent" - which gives an awful cache hit rate - since
>> proxy_cache takes this into account, unless I use something like
>> "proxy_ignore_headers Vary;"
>>
>> But ignoring Vary headers can cause other issues such as gzipped content
>> being sent to a non-gzip client.
>>
>> So I'm looking for a way to basically rewrite the vary header to "Vary:
>> Accept-Encoding" before storing it in proxy_cache - but I wonder if this is
>> even possible in nginx, and if yes - can you give any pointers?
>>
>> I found a temporary fix, and that is to ignore the Vary header, and using a
>> custom variable as a part of the cache key, that is either "", "gzip" or
>> "deflate" (I use a map to look at the Accept-Encoding header from the
>> client).
>>
>> This works great - but I rather keep the cache key a bit clean (since I'll
>> use it later)
>>
>> Do you guys have any recommendations how to make this happen?
>
> The best possible solution I can think of is to ask the client to fix
> the Vary header it returns.  Using User-Agent in Vary is something
> one shouldn't use without a very good reason, and if there a
> reason - it's likely a bad idea to strip from the Vary header.
> And if there are no reasons, then it shouldn't be returned in the
> first place.
>
>> Also as a side note, if I remove the custom variable from the cache key,
>> how would one actually purge the file then? I assume I have to send
>> different purge requests, since the cached file is based on the Vary:
>> accept-encoding - so I'd have to purge at least the amount of cached
>> encodings right?
>
> When using purge as availalbe in nginx-plus
> (http://nginx.org/r/proxy_cache_purge), it takes care of removing
> all cached variants, much like it does for wildcard purge requests.
>
>> Also I could opt for another way, and that's always requesting a
>> uncompressed file from the origin (Is it simply not sending the
>> accept-encoding header, or should I do something else?), and then on every
>> request either decide to gzip it or not - the downside I see here, is the
>> fact that most clients request gzip,deflate content, so having to compress
>> on every request will use additional CPU resources.
>
> This can be done easily, just
>
>      proxy_set_header Accept-Encoding "";
>
> should be enough.  Alternatively, you can use
>
>      proxy_set_header Accept-Encoding gzip;
>      gunzip on;
>
> to always ask gzipped resources and gunzip them when needed, see
> http://nginx.org/en/docs/http/ngx_http_gunzip_module.html.
>