Nginx stuck processes

Tue Oct 18 10:49:09 UTC 2011

On Tue, Oct 18, 2011 at 12:31:49PM +0200, Sandesh K. wrote:
> Hi,
> 
> We are using nginx version: 1.0.0 for pop-proxy. We have seen off-late
> certain processes remaining in state whereby nginx becomes
> non-responsive.
> 
> Below is the dump of ps showing the stuck process of nginx ==> R -->
> 4150:24.
> 
> i have shared the gdb dump and the backtrace of the stuck process.
> 
> 38487 ?        S      0:00 nginx: master process
> /usr/local/nginx/sbin/nginx -c /etc/nginx/conf/nginx.conf
>  7044 ?        R    4150:24  \_ nginx: worker process
> 22408 ?        S      0:03 /usr/local/apache3/sbin/httpd
> 22461 ?        S      4:25  \_ /usr/local/apache3/sbin/httpd
> 27699 ?        D      0:00  |   \_
> /home/httpd/dal/cgi-bin/dal/rmailajax.cgi
> 15853 ?        S      0:06  \_ /usr/local/apache3/sbin/httpd
> 17290 ?        S      0:05  \_ /usr/local/apache3/sbin/httpd
> 22796 ?        S      0:04 /usr/sbin/httpd -DSSL
> 22801 ?        S      0:29  \_ /usr/sbin/httpd -DSSL
> 20605 ?        S      4:05  \_ /usr/sbin/httpd -DSSL
> 19391 ?        S      2:33  \_ /usr/sbin/httpd -DSSL
> 60469 ?        S      0:59  \_ /usr/sbin/httpd -DSSL
>  3437 ?        S      0:35  \_ /usr/sbin/httpd -DSSL
> 11200 ?        S      0:30  \_ /usr/sbin/httpd -DSSL
> 16597 ?        S      0:19  \_ /usr/sbin/httpd -DSSL
> 19556 ?        S      0:09  \_ /usr/sbin/httpd -DSSL
> 27218 ?        S      0:00  \_ /usr/sbin/httpd -DSSL
> 
> [root at xxx logs]# strace
> [root at xxx logs]# gdb
> GNU gdb Red Hat Linux (6.3.0.0-0.30.1rh)
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you
> are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for
> details.
> This GDB was configured as "i386-redhat-linux-gnu".
> (gdb) attach 7044
> Attaching to process 7044
> Reading symbols from /usr/local/nginx/sbin/nginx...done.
> Using host libthread_db library "/lib/tls/libthread_db.so.1".
> Reading symbols from /lib/libcrypto.so.4...done.
> Loaded symbols for /lib/libcrypto.so.4
> Reading symbols from /lib/tls/libc.so.6...done.
> Loaded symbols for /lib/tls/libc.so.6
> Reading symbols from /usr/kerberos/lib/libgssapi_krb5.so.2...done.
> Loaded symbols for /usr/kerberos/lib/libgssapi_krb5.so.2
> Reading symbols from /usr/kerberos/lib/libkrb5.so.3...done.
> Loaded symbols for /usr/kerberos/lib/libkrb5.so.3
> Reading symbols from /usr/kerberos/lib/libcom_err.so.3...done.
> Loaded symbols for /usr/kerberos/lib/libcom_err.so.3
> Reading symbols from /usr/kerberos/lib/libk5crypto.so.3...done.
> Loaded symbols for /usr/kerberos/lib/libk5crypto.so.3
> Reading symbols from /lib/libresolv.so.2...done.
> Loaded symbols for /lib/libresolv.so.2
> Reading symbols from /lib/libdl.so.2...done.
> Loaded symbols for /lib/libdl.so.2
> Reading symbols from /usr/lib/libz.so.1...done.
> Loaded symbols for /usr/lib/libz.so.1
> Reading symbols from /lib/ld-linux.so.2...done.
> Loaded symbols for /lib/ld-linux.so.2
> Reading symbols from /lib/libnss_files.so.2...done.
> Loaded symbols for /lib/libnss_files.so.2
> 0x001dbb74 in _int_malloc () from /lib/tls/libc.so.6
> (gdb) backtrace
> #0  0x001dbb74 in _int_malloc () from /lib/tls/libc.so.6
> #1  0x001dc89c in _int_memalign () from /lib/tls/libc.so.6
> #2  0x001db2c7 in memalign () from /lib/tls/libc.so.6
> #3  0x001dd367 in posix_memalign () from /lib/tls/libc.so.6
> #4  0x0805d614 in ngx_memalign (alignment=16, size=2746520, log=0x108)
> at src/os/unix/ngx_alloc.c:56
> #5  0x0804b603 in ngx_create_pool (size=256, log=0x93bd1a0) at
> src/core/ngx_palloc.c:20
> #6  0x0805be26 in ngx_event_accept (ev=0x93df338) at
> src/event/ngx_event_accept.c:124
> #7  0x0805b7f9 in ngx_event_process_posted (cycle=0x93bce28,
> posted=0x8070e3c) at src/event/ngx_event_posted.c:39
> #8  0x0805a8e1 in ngx_process_events_and_timers (cycle=0x93bce28) at
> src/event/ngx_event.c:253
> #9  0x0806051a in ngx_worker_process_cycle (cycle=0x93bce28, data=0x0)
> at src/os/unix/ngx_process_cycle.c:800
> #10 0x0805eb3c in ngx_spawn_process (cycle=0x93bce28, proc=0x8060465
> <ngx_worker_process_cycle>, data=0x0, name=0x806b971 "worker process",
> respawn=17)
>     at src/os/unix/ngx_process.c:196
> #11 0x0806023e in ngx_reap_children (cycle=0x93bce28) at
> src/os/unix/ngx_process_cycle.c:617
> #12 0x0805f8e1 in ngx_master_process_cycle (cycle=0x93bce28) at
> src/os/unix/ngx_process_cycle.c:180
> #13 0x08049fea in main (argc=3, argv=0x93bd5a4) at src/core/nginx.c:405

It seems that malloc() loops inside. This may be bug in nginx or glibc.
Could you show:

nginx -V
uname -osrvmpi

-- 
Igor Sysoev