Limiting total disk space used by cache + temp files

[Maxim Dounin]

Hello!

On Thu, May 04, 2017 at 02:50:53AM -0400, Michael Corn wrote:

> I'm trying to limit the overall disk usage for the cache + any temp files.
> 
> I have specified:
>                         proxy_cache_path  /tmp/cache max_size=50m 
> use_temp_path=off;
>                         proxy_max_temp_file_size 1m;
> 
> Yet, what I observe is that the temp file will grow to the full size of the
> file being retrieved from the upstream, ignoring the value of
> proxy_max_temp_file_size, and ignoring the max_size for the cache. Only
> after the file is brought over completely does the cache manager kick in and
> delete something older from the cache to bring it back in line with the 50mb
> limit.
> 
> For example, if I had in the cache file1 (20 MB) and file2 (20 MB), and I
> receive a request for file3 which is 20 MB, the total size of the /tmp/cache
> directory will grow to 60 MB. Only after file3 is received in its entirety
> will file1 be deleted bringing the size back down to 40 MB.
> 
> I don't mind setting use_temp_path=on, but in that case as well the temp
> file grows beyond the proxy_max_temp_file_size.
> 
> Any advice?

What you are trying to do is not going to work.

First of all, proxy_max_temp_file_size is ignored when using 
cache.  Quoting http://nginx.org/r/proxy_max_temp_file_size:

: This restriction does not apply to responses that will be cached 
: or stored on disk.

As for the cache max_size parameter, it is maintained by the cache 
manager process, and it does so after the files are already in 
the cache.  Quoting http://nginx.org/r/proxy_cache_path:

: The special "cache manager" process monitors the maximum cache 
: size set by the max_size parameter. When this size is exceeded, it 
: removes the least recently used data.

Moreover, cache manager may be busy with other tasks or just 
sleeping, so there is inevitable difference between max_size 
configured and possible maximum size of the cache.

-- 
Maxim Dounin
http://nginx.org/