0.8.52 AIO problem

[Brane F. Gračnar]

On Thursday 07 of October 2010 13:44:58 Maxim Dounin wrote:
> You may want to try to reproduce the problem without third party
> modules, but most likely it's problem in nginx itself.  AFAIK aio
> support still has some socket leaks.  Probably Igor knows more
> details.
> 
> If you feel yourself brave enough and want to debug this problem -
> you may recompile nginx --with-debug, configure debug logging,
> enable core dumps and add "debug_points abort;" to your
> nginx.conf.  This will cause abort() to be called when nginx
> detects open sockets on worker process shutdown (and hence core
> dump), and will allow to carefully examine connections with open
> sockets left.

I recompiled nginx without 3rd party modules and with debug enabled.

nginx -V
--- snip ---
nginx version: nginx/0.8.52
TLS SNI support enabled
configure arguments: --with-cc-opt='-O2 -pipe -fomit-frame-pointer -funroll-loops -g' --prefix=/usr --conf-path=/etc/nginx/nginx.conf --error-log-
path=/tmp/nginx.error --http-log-path=/tmp/nginx.access --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --user=daemon --group=daemon --http-
client-body-temp-path=/var/spool/nginx/client_body_temp --http-proxy-temp-path=/var/spool/nginx/proxy_temp --http-fastcgi-temp-
path=/var/spool/nginx/fastcgi_temp --with-ipv6 --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_dav_module --with-
http_flv_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-mail 
--with-mail_ssl_module --with-pcre --with-file-aio --with-debug
--- snip ---

The problem persists. See attachments.

Nginx crashes/aborts with messages like "[alert] 15248#0: open socket #491 left in connection 217" only if i reload nginx configuration (kill -HUP).

Stacktrace:

--- snip ---
#0  0x00007f8687407a75 in raise () from /lib/libc.so.6
(gdb) bt
#0  0x00007f8687407a75 in raise () from /lib/libc.so.6
#1  0x00007f868740b5c0 in abort () from /lib/libc.so.6
#2  0x000000000042ae95 in ngx_debug_point () at src/os/unix/ngx_process.c:564
#3  0x000000000042d07a in ngx_worker_process_exit (cycle=0x1265c40) at src/os/unix/ngx_process_cycle.c:1039
#4  0x000000000042d15e in ngx_worker_process_cycle (cycle=0x1265c40, data=<value optimized out>) at src/os/unix/ngx_process_cycle.c:789
#5  0x000000000042b0a4 in ngx_spawn_process (cycle=0x1265c40, proc=<value optimized out>, data=<value optimized out>, name=0x49b82e "worker process", 
respawn=-3) at src/os/unix/ngx_process.c:196
#6  0x000000000042c40e in ngx_start_worker_processes (cycle=0x1265c40, n=8, type=-3) at src/os/unix/ngx_process_cycle.c:355
#7  0x000000000042d6d5 in ngx_master_process_cycle (cycle=0x1265c40) at src/os/unix/ngx_process_cycle.c:136
#8  0x00000000004070d7 in main (argc=<value optimized out>, argv=<value optimized out>) at src/core/nginx.c:401
--- snip ---

Btw, everything works ok if aio and directio are disabled.

Best regards, Brane
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nginx_aio_opened_filedescriptors.png
Type: image/png
Size: 42754 bytes
Desc: nginx_aio_opened_filedescriptors.png
URL: <http://nginx.org/pipermail/nginx/attachments/20101007/6ea2b3f2/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nginx_aio_status_rate.png
Type: image/png
Size: 33291 bytes
Desc: nginx_aio_status_rate.png
URL: <http://nginx.org/pipermail/nginx/attachments/20101007/6ea2b3f2/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nginx_aio_status_clients.png
Type: image/png
Size: 30435 bytes
Desc: nginx_aio_status_clients.png
URL: <http://nginx.org/pipermail/nginx/attachments/20101007/6ea2b3f2/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nginx_aio_vmstat_memory.png
Type: image/png
Size: 47782 bytes
Desc: nginx_aio_vmstat_memory.png
URL: <http://nginx.org/pipermail/nginx/attachments/20101007/6ea2b3f2/attachment-0007.png>