Cache manager occasionally stops deleting cached files

[vergil]

Hello again.

Maxim Dounin Wrote:
-------------------------------------------------------
> Hello!
> 
> On Thu, Feb 25, 2016 at 05:46:32AM -0500, vergil wrote:
> 
> > vedranf Wrote:
> > -------------------------------------------------------
> > > Hello,
> > > 
> > > I'm having an issue where nginx (1.8) cache manager suddenly just
> > > stops deleting content thus the disk soon ends up being full until
> I
> > > restart it by hand. After it is restarted, it works normally for a
> > > couple of days, but then it happens again. Cache has some 30-40k
> > > files, nothing huge. Relevant config lines are:
> 
> [...]
> 
> > We have the same problem, but i'm not sure, that this is caused by
> often
> > nginx restarts.
> 
> This particular case was traced to segmentation faults, likely 
> caused by 3rd party modules.
> 
> [...]
> 
> > Also, i think it's somehow related to write connection leak. (see
> image
> > link)
> > 
> >
> https://s3.eu-central-1.amazonaws.com/drive-public-eu/nginx/betelgeuse
> _nginx_connections.PNG
> 
> [...]
> 
> > As you see write connections continuously grows. (When we had to
> power off
> > the machine it's reached ~60k).
> > 
> > For counting nginx connections we use standard
> http_stub_status_module.
> > 
> > I think that nginx "reference counter" could be broken, because
> total
> > established TCP connection remains the same all the time.
> 
> Writing connections will grow due to segmentation faults as well, 
> so you are likely have the same problem.  See basic 
> recommendations in my initial answer in this threads.

I've maded custom nginx build using latest version (1.9.13) without 3rd
party modules:

nginx -V
nginx version: nginx/1.9.13
built by gcc 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.1)
built with OpenSSL 1.0.1f 6 Jan 2014
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -fPIE -fstack-protector
--param=ssp-buffer-size=4 -Wformat -Werror=format-security
-D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -fPIE -pie
-Wl,-z,relro -Wl,-z,now' --prefix=/usr/share/nginx
--conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log
--error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock
--pid-path=/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/body
--http-fastcgi-temp-path=/var/lib/nginx/fastcgi
--http-proxy-temp-path=/var/lib/nginx/proxy
--http-scgi-temp-path=/var/lib/nginx/scgi
--http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit
--with-ipv6 --with-http_ssl_module --with-http_stub_status_module
--with-http_realip_module --with-http_auth_request_module
--with-http_addition_module --with-http_dav_module --with-http_gunzip_module
--with-http_gzip_static_module --with-http_v2_module --with-http_sub_module
--with-http_xslt_module --with-stream --with-stream_ssl_module --with-mail
--with-mail_ssl_module --with-threads

Nothing changed: connections continuously grow, cache manager works fine and
not filled the disk yet, but i think it's a matter of 2-3 days.

PIDs didn't changed since the start, and log didn't contain "worker process
exited ..." messages.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,264599,266142#msg-266142