Nginx proxy cache purge process does not clean up items fast enough for new elements

[Maxim Dounin]

Hello!

On Wed, Oct 16, 2019 at 09:44:08AM -0400, sachin.shetty at gmail.com wrote:

> Thankyou Maxim, is there anyway I can make the cache manager a bit more
> aggressive in prune and purge? We already leave 20%  of space free on the
> disks, but the concurrent request rate for large files can be huge and we
> still run in to this issue. 

Most likely you are hitting space limits due to temporary files, which 
aren't managed by cache manager at all.  My recommendation would 
be to consider proxy_cache_lock and/or "proxy_cache_use_stale 
updating" to reduce number of concurrent requests trying to cache 
the same files, see here:

http://nginx.org/r/proxy_cache_lock
http://nginx.org/r/proxy_cache_use_stale

Also you may want to tune things like proxy_cache_min_uses to 
reduce unneeded caching on sporadic requests.

> What are your thoughts about disabling buffering on such issues? This is not
> a fatal error, so we should stop buffering and switch to streaming mode and
> let the request succeed with a error log line in error.log. 

Yes, in theory.  But the question is how complex and error prone 
the resulting code would be.  Hence the current choice is to treat 
such errors as fatal.

-- 
Maxim Dounin
http://mdounin.ru/