reverse proxy and too many temp files

[Maxim Dounin]

Hello!

On Wed, Oct 27, 2010 at 05:16:02PM -0400, roadfox wrote:

> hi
> 
> we are using nginx as a reverse proxy in front of some webservers with
> media files.
> evrerithing works great beside when we start nginx new with an empty
> cache.
> 
> in this situation allot of files in proxy_temp_path are create consuming
> quite fast all our diskspace, in the same time proxy_cache_path is only
> very slowly populated.
> 
> i see two parameters who might help:
> proxy_cache_use_stale 
> proxy_max_temp_file_size
> 
> a quick test with *use_stale was not succesfull, but maybe something
> else was wrong

proxy_cache_use_stale only make sense if you already have data in 
cache.

> and for proxy_max_temp_file_size=0 i'm not shure if it is the right
> thing todo

proxy_max_temp_file_size can't (and doesn't) limit writing to disk 
responses expected to be saved to cache as nginx needs them on 
disk to save to cache.

> what we would like to achieve is to uses as much of the diskspace for
> caching. but on a restart with clean cache, not filling up the temp
> folder which does not count to proxy_cache_path max_size.
> 
> or should we just reduce the workers (16) to provent from having to many
> workers creating the same temp file for the same origin file?

No, reducing number of workers isn't likely to help as each worker 
may handle many connections.

Solution to this problem is so called "busy locks" which are not 
yet implemented.

Right now there is no good solution.  Some possibilities include: 
using limit_zone to limit number of requests (though this will 
limit serving from cache as well), warming up cache before 
startup, adding more disk space.

Maxim Dounin