0.8.52 AIO problem

[Maxim Dounin]

Hello!

On Thu, Oct 07, 2010 at 12:11:59PM +0200, Brane F. Gračnar wrote:

> Hello!
> 
> I'm using nginx for serving static content. Nginx is serving from a folder 
> containing millions of small files.
> 
> Becouse machine doesn't have lots of ram (2G) and becouse access pattern is 
> completely random VFS cache gets trashed.
> 
> That's why i tried aio + directio.
> 
> Configuration:
> 
> location / {
> 	aio on;
>    directio 1;
> }
> 
> It seems that everything works fine, vmstat reports smaller cache size, etc. 
> But number of opened filedescriptors on system grows and nginx status module 
> reports more and more connections in state "writing".
> 
> Nginx main error contains tons of familiar messages:
> 
> --- snip ---
> 2010/10/06 17:48:42 [alert] 1589#0: open socket #307 left in connection 128
> 2010/10/06 17:48:42 [alert] 1589#0: open socket #313 left in connection 131
> 2010/10/06 17:48:42 [alert] 1589#0: open socket #297 left in connection 132
> 2010/10/06 17:48:42 [alert] 1589#0: open socket #312 left in connection 133
> 2010/10/06 17:48:42 [alert] 1589#0: open socket #364 left in connection 134
> 2010/10/06 17:48:42 [alert] 1589#0: open socket #394 left in connection 136
> 2010/10/06 17:48:42 [alert] 1589#0: open socket #321 left in connection 138
> 2010/10/06 17:48:42 [alert] 1589#0: open socket #320 left in connection 139
> 2010/10/06 17:48:42 [alert] 1589#0: open socket #260 left in connection 141
> 2010/10/06 17:48:42 [alert] 1589#0: open socket #336 left in connection 145
> 2010/10/06 17:48:42 [alert] 1589#0: open socket #340 left in connection 148
> 2010/10/06 17:48:42 [alert] 1589#0: open socket #347 left in connection 149
> --- snip ---
> 
> Am i doing something wrong?
> 
> nginx version: nginx/0.8.52
> TLS SNI support enabled
> configure arguments: --with-cc-opt='-O2 -pipe -fomit-frame-pointer -funroll-
> loops' --prefix=/usr --conf-path=/etc/nginx/nginx.conf --error-log-
> path=/tmp/nginx.error --http-log-path=/tmp/nginx.access --pid-
> path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --user=daemon --
> group=daemon --http-client-body-temp-path=/var/spool/nginx/client_body_temp --
> http-proxy-temp-path=/var/spool/nginx/proxy_temp --http-fastcgi-temp-
> path=/var/spool/nginx/fastcgi_temp --with-ipv6 --with-http_ssl_module --with-
> http_realip_module --with-http_addition_module --with-http_dav_module --with-
> http_flv_module --with-http_gzip_static_module --with-http_random_index_module 
> --with-http_secure_link_module --with-http_stub_status_module --with-mail --
> with-mail_ssl_module --with-pcre --with-file-aio --add-
> module=/export/tmp/pkgcompile/nginx/work/nginx-0.8.52/modules/nginx_mod_h264_streaming-2.2.7 
> --add-module=/export/tmp/pkgcompile/nginx/work/nginx-0.8.52/modules/ngx-
> fancyindex --add-
> module=/export/tmp/pkgcompile/nginx/work/nginx-0.8.52/modules/ngx_http_log_request_speed
> 
> Linux 2.6.35, x86_64, Ubuntu 10.04LTS.

You may want to try to reproduce the problem without third party 
modules, but most likely it's problem in nginx itself.  AFAIK aio 
support still has some socket leaks.  Probably Igor knows more 
details.

If you feel yourself brave enough and want to debug this problem - 
you may recompile nginx --with-debug, configure debug logging, 
enable core dumps and add "debug_points abort;" to your 
nginx.conf.  This will cause abort() to be called when nginx 
detects open sockets on worker process shutdown (and hence core 
dump), and will allow to carefully examine connections with open 
sockets left.

See here for basic debugging instructions:
http://wiki.nginx.org/Debugging

Maxim Dounin