Worker processes in hung state after reload

[Maxim Dounin]

Hello!

On Fri, Jul 01, 2011 at 08:01:11PM -0400, adam wrote:

> We use Nginx as a reverse proxy.  After 2 or 3 reloads (kill -HUP), the
> parent's one child worker process remains "stuck"; the parent spawns a
> new one which is also often "stuck."
> 
> The number of worker_processes we have configured is 1. After a reload,
> there are 2 workers in the process list. After another reload, 3
> workers, and so on. Usually all of these workers are in state R. When we
> strace them, we see no system calls, but they are sucking on user CPU.
> They are unable to serve requests. One time (seen in detail below), a
> new worker was spawned and it was *not* in state R, but state S, and was
> able to serve requests, until we issued another reload.
> 
> Of course what should happen is, after a reload, we should only ever
> have one worker process which is able to serve requests.
> 
> Details below, including our config. We can provide lsof, strace, debug
> log of the reproduction case if needed.
> 
> #############
> 
> 07/01 23:10[root at proxy ~]# cat /etc/redhat-release 
> CentOS release 5.6 (Final)
> 
> 07/01 23:10[root at proxy ~]# uname -a
> Linux proxy 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686
> i686 i386 GNU/Linux
> 
> 07/01 23:10[root at proxy ~]# nginx -V
> nginx version: nginx/0.8.54
> built by gcc 4.1.2 20080704 (Red Hat 4.1.2-48)
> TLS SNI support disabled
> configure arguments: --user=nginx --group=nginx
> --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx
> --conf-path=/etc/nginx/nginx.conf
> --error-log-path=/var/log/nginx/error.log
> --http-log-path=/var/log/nginx/access.log
> --http-client-body-temp-path=/var/lib/nginx/tmp/client_body
> --http-proxy-temp-path=/var/lib/nginx/tmp/proxy
> --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi
> --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi
> --http-scgi-temp-path=/var/lib/nginx/tmp/scgi
> --pid-path=/var/run/nginx.pid --lock-path=/var/lock/subsys/nginx
> --with-http_ssl_module --with-http_realip_module
> --with-http_addition_module --with-http_xslt_module
> --with-http_image_filter_module --with-http_sub_module
> --with-http_gzip_static_module --with-http_random_index_module
> --with-http_secure_link_module --with-http_degradation_module
> --with-http_stub_status_module --with-http_perl_module --with-ipv6
> --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
> -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386
> -mtune=generic -fasynchronous-unwind-tables' --with-cc-opt='-O2 -g -pipe
> -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
> --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic
> -fasynchronous-unwind-tables'
> --add-module=src/http/modules/nginx_syslog_patch
> --add-module=src/http/modules/nginx_upstream_module
> --add-module=src/http/modules/nginx_ajp_module
> 
> (Note above that we have a few third-party modules added for syslog and
> upstream health checking.)

Are you able to reproduce the problem without third party 
modules/patches and with latest stable release (1.0.4)?

Maxim Dounin