0.8.52 AIO problem

[Maxim Dounin]

Hello!

On Thu, Oct 07, 2010 at 04:28:30PM +0200, Brane F. Gračnar wrote:

> On Thursday 07 of October 2010 13:44:58 Maxim Dounin wrote:
> > You may want to try to reproduce the problem without third party
> > modules, but most likely it's problem in nginx itself.  AFAIK aio
> > support still has some socket leaks.  Probably Igor knows more
> > details.
> > 
> > If you feel yourself brave enough and want to debug this problem -
> > you may recompile nginx --with-debug, configure debug logging,
> > enable core dumps and add "debug_points abort;" to your
> > nginx.conf.  This will cause abort() to be called when nginx
> > detects open sockets on worker process shutdown (and hence core
> > dump), and will allow to carefully examine connections with open
> > sockets left.
> 
> I recompiled nginx without 3rd party modules and with debug enabled.
> 
> nginx -V
> --- snip ---
> nginx version: nginx/0.8.52
> TLS SNI support enabled
> configure arguments: --with-cc-opt='-O2 -pipe -fomit-frame-pointer -funroll-loops -g' --prefix=/usr --conf-path=/etc/nginx/nginx.conf --error-log-
> path=/tmp/nginx.error --http-log-path=/tmp/nginx.access --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --user=daemon --group=daemon --http-
> client-body-temp-path=/var/spool/nginx/client_body_temp --http-proxy-temp-path=/var/spool/nginx/proxy_temp --http-fastcgi-temp-
> path=/var/spool/nginx/fastcgi_temp --with-ipv6 --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_dav_module --with-
> http_flv_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-mail 
> --with-mail_ssl_module --with-pcre --with-file-aio --with-debug
> --- snip ---
> 
> The problem persists. See attachments.

Ok, that was expected.

> Nginx crashes/aborts with messages like "[alert] 15248#0: open 
> socket #491 left in connection 217" only if i reload nginx 
> configuration (kill -HUP).

That's expected too, as this check only happens on worker exit.

> Stacktrace:
> 
> --- snip ---
> #0  0x00007f8687407a75 in raise () from /lib/libc.so.6
> (gdb) bt
> #0  0x00007f8687407a75 in raise () from /lib/libc.so.6
> #1  0x00007f868740b5c0 in abort () from /lib/libc.so.6
> #2  0x000000000042ae95 in ngx_debug_point () at src/os/unix/ngx_process.c:564
> #3  0x000000000042d07a in ngx_worker_process_exit (cycle=0x1265c40) at src/os/unix/ngx_process_cycle.c:1039
> #4  0x000000000042d15e in ngx_worker_process_cycle (cycle=0x1265c40, data=<value optimized out>) at src/os/unix/ngx_process_cycle.c:789
> #5  0x000000000042b0a4 in ngx_spawn_process (cycle=0x1265c40, proc=<value optimized out>, data=<value optimized out>, name=0x49b82e "worker process", 
> respawn=-3) at src/os/unix/ngx_process.c:196
> #6  0x000000000042c40e in ngx_start_worker_processes (cycle=0x1265c40, n=8, type=-3) at src/os/unix/ngx_process_cycle.c:355
> #7  0x000000000042d6d5 in ngx_master_process_cycle (cycle=0x1265c40) at src/os/unix/ngx_process_cycle.c:136
> #8  0x00000000004070d7 in main (argc=<value optimized out>, argv=<value optimized out>) at src/core/nginx.c:401
> --- snip ---

Backtrace is mostly useless here as it shows trace to check, not 
to the real problem.  Though it shows that you may want to 
recompile again with lower optimization to simplify debugging 
(note this "<value optimized out>" things).  I usually debug 
with -O0 as it minimizes compiler artifacts.  For CPU-bound hosts 
you may want to use -O1 instead.

As I already said, you have to examine connections with open 
sockets left.  Something like this in gdb should be usefull 
(assuming your error message with connection 217 matches core):

p ngx_cycle->connections[217]
p ngx_cycle->connections[217].log->connection
p *((ngx_http_request_t *) ngx_cycle->connections[217].data)

In debug log records with with *<number> should be for the 
connection in question, where <number> is one from 2nd gdb command 
(log->connection).

And, BTW, you may want to show your config if you aren't going to 
debug it yourself but asking for help instead. ;)

Maxim Dounin