massive deleted open files in proxy cache

[Marco Lenhardt]

Hi!

We are useing Ubuntu 16.04 with nginx version 1.10.0-0ubuntu0.16.04.4.

nginx.conf:

user  nginx;
worker_processes  auto;
worker_rlimit_nofile 20480; # ulimit open files per worker process

events {
         # Performance
         worker_connections  2048; # openfilelimits beachten
         multi_accept on;
         use epoll;
}

http {
         open_file_cache max=10000 inactive=1d;
         open_file_cache_valid 1d;
         open_file_cache_min_uses 1;
         open_file_cache_errors off;

         proxy_cache_path /var/cache/nginx/proxy_cache 
levels=1:2 keys_zone=html_cache:30m max_size=8192m inactive=4h 
use_temp_path=off;
         proxy_cache_path /var/cache/nginx/wordpress_cache 
levels=1:2 keys_zone=wordpress_cache:1m max_size=256m inactive=24h 
use_temp_path=off;

         proxy_buffer_size          128k;
         proxy_buffers              4 256k;
         proxy_busy_buffers_size    256k;
}


# df -h |grep cache
tmpfs      300M   74M  227M  25% /var/cache/nginx/wordpress_cache
tmpfs      9.0G  4.1G  5.0G  45% /var/cache/nginx/proxy_cache

# df -i |grep cache
tmpfs    20632978    5457 20627521    1% /var/cache/nginx/wordpress_cache
tmpfs    20632978   74613 20558365    1% /var/cache/nginx/proxy_cache

# grep cache /etc/fstab
tmpfs	/var/cache/nginx/proxy_cache/		tmpfs 
rw,uid=109,gid=117,size=9G,mode=0755	0 0
tmpfs	/var/cache/nginx/wordpress_cache/	tmpfs 
rw,uid=109,gid=117,size=300M,mode=0755	0 0


# free -m
               total        used        free      shared  buff/cache 
available
Mem:         161195      112884        1321        4626       46988 
  42519
Swap:          3903         211        3692


Problem:
========
We got massive open file handles from nginx user located inside 
proxy_cache_path with status deleted:

# lsof -E -M -T > lsof.`date +"%Y%d%m-%H%M%S"`.out

nginx  3613  nginx   48r   REG   0,42  148664   29697 
/var/cache/nginx/proxy_cache/temp/5/23/04ca8002edd2daa3c538ada5b202d6eb 
(deleted)
nginx  3613  nginx   50r   REG   0,42  161618   19416 
/var/cache/nginx/proxy_cache/temp/1/40/d8f0a3563d18af4fbf43242e19283b15 
(deleted)

# grep nginx lsof.20172002-085328.out |wc -l
69003

# grep nginx lsof.20172002-085328.out |grep deleted |wc -l
36312

# grep nginx lsof.20172002-085328.out |grep deleted |grep 
"/var/cache/nginx/proxy_cache/temp" |wc -l
32004

The most of the 36k deleted files are located inside the temp cache folder.

My question is, why we got so much deleted files inside the cache. Why 
is nginx not freeing these files?
Is there a problem with the proxy_cache_path option use_temp_path=off ?

I am in worry that the cache file system will be filled up with deleted 
files, and will reach open files limits.

Or do we have a nginx misconfiguration somewhere?

Additionally we were often visited from the oom-killer, (in face of 20G 
free memory). If I restart nginx before we reach 80k open ngnix files, 
the oom-killer will not visit us!

Has anybody simmilar findings regarding deleted open files?

br,
Marco