Too many deleted open files in proxy_temp_path

[Maxim Dounin]

Hello!

On Fri, Apr 10, 2020 at 11:26:17AM -0400, patademahesh wrote:

> We are using nginx as reverse proxy to cache static content for a moodle LMS
> site. The caching part is working fine but we started facing the cache path
> disk full issues. When we checked using du, it was reporting very low used
> space.Then we checked lsof output we found too many deleted file entries. We
> realized, that file gets deleted but it holds the FD and disk space is never
> released until we restart the nginx server. Most of the files were from
> proxy_temp_path location. We tried this with tmpfs and normal disk but the
> end result was same.
> 
> # nginx -v
> nginx version: nginx/1.10.3 (Ubuntu)

Note that when writing temporary files when proxying (without 
caching, but there is no caching configured in "location /"), it 
is quite normal that temporary files are unlinked (deleted) right 
after creation and cannot be seen by "du".  This way temporary 
files are automatically removed by the system as long as a file is 
closed - or if nginx is killed or crashes.

If you think that temporary files are not removed from disk even 
if corresponding client connections are closed - this might 
indicate a socket leak.  Usually, socket leaks can be seen by 
other symptoms as well - such as connections in the CLOSED state 
as shown by "netstat -an", or "open socket ... left in connection ..." 
alerts during graceful shutdown of worker processes.  Given 
that you are using an ancient nginx version, socket leaks might be 
the case - there are at least some fixed since 1.10.3, mostly in 
HTTP/2.

On the other hand, if deleted files is the only symptom you are 
seeing, it might simply indicate that corresponding client 
connection is still open and the client is slowly downloading the 
response in question.  If that's true, you may want to either use 
a filesystem with more space for temporary files, or tune 
proxy buffering, see here:

http://nginx.org/r/proxy_max_temp_file_size

-- 
Maxim Dounin
http://mdounin.ru/