nginx worker process hang,cpu load 100%

Long Wan nginx-forum at nginx.us
Mon Nov 14 14:37:29 UTC 2011


Hello,Maxim. Thanks for you help.

I reproduce the problem in nginx-1.0.9,

[root at host-22 ~]# /usr/local/nginx/sbin/nginx -V
nginx: nginx version: nginx/1.0.9
nginx: built by gcc 4.1.2 20080704 (Red Hat 4.1.2-51)
nginx: TLS SNI support disabled
nginx: configure arguments: --user=www --group=www
--prefix=/usr/local/nginx --with-http_stub_status_module
--with-http_ssl_module --with-openssl-opt=enable-tlsext
--with-http_sub_module --with-cc-opt=-O2 --with-cpu-opt=opteron
--add-module=../ngx_cache_purge-1.4
[root at host-22 ~]# 
[root at host-22 ~]# 
[root at host-22 ~]# ps aux|grep -e CPU -e nginx
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME
COMMAND
root       569  0.0  0.0  49016  6884 ?        Ss   21:59   0:00 nginx:
master process /usr/local/nginx/sbin/nginx
www        587 97.3  0.0  49380  7556 ?        R    22:00  10:16 nginx:
worker process      
www        588 93.8  0.0  49380  7556 ?        R    22:00   9:54 nginx:
worker process      
www        614 43.1  0.0  49412  7584 ?        T    22:01   4:07 nginx:
worker process      
root       781  0.0  0.1  95984 19684 pts/0    S+   22:06   0:00 gdb
/usr/local/nginx/sbin/nginx 614
www        876  0.0  0.0  50504  8464 ?        S    22:10   0:00 nginx:
worker process      
www        877  0.0  0.0  50504  8464 ?        S    22:10   0:00 nginx:
worker process      
www        878  0.0  0.0  50504  8464 ?        S    22:10   0:00 nginx:
worker process      
www        879  0.0  0.0  50504  8660 ?        S    22:10   0:00 nginx:
worker process      
www        880  0.0  0.0  50504  8464 ?        S    22:10   0:00 nginx:
worker process      
www        881  0.0  0.0  50504  8464 ?        S    22:10   0:00 nginx:
worker process      
www        882  0.0  0.0  50504  8464 ?        S    22:10   0:00 nginx:
worker process      
www        883  0.0  0.0  50504  8464 ?        S    22:10   0:00 nginx:
worker process      
root       954  0.0  0.0  61168   788 pts/1    S+   22:10   0:00 grep -e
CPU -e nginx
[root at host-22 ~]# 



[root at host-22 ~]# gdb /usr/local/nginx/sbin/nginx 614
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show
copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/nginx/sbin/nginx...done.
Attaching to program: /usr/local/nginx/sbin/nginx, process 614
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libpcre.so.0...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libpcre.so.0
Reading symbols from /lib64/libssl.so.6...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libssl.so.6
Reading symbols from /lib64/libcrypto.so.6...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libcrypto.so.6
Reading symbols from /lib64/libdl.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /usr/lib64/libz.so.1...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libz.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/libgssapi_krb5.so.2...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libkrb5.so.3...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /usr/lib64/libk5crypto.so.3...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libk5crypto.so.3
Reading symbols from /usr/lib64/libkrb5support.so.0...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libsepol.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libsepol.so.1
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libnss_files.so.2
ngx_http_upstream_get_peer (pc=0x54ae260, data=<value optimized out>) at
src/http/ngx_http_upstream_round_robin.c:632
632             if (reset++) {
(gdb) bt
#0  ngx_http_upstream_get_peer (pc=0x54ae260, data=<value optimized
out>) at src/http/ngx_http_upstream_round_robin.c:632
#1  ngx_http_upstream_get_round_robin_peer (pc=0x54ae260, data=<value
optimized out>) at src/http/ngx_http_upstream_round_robin.c:425
#2  0x000000000041a99c in ngx_event_connect_peer (pc=0x54ae960) at
src/event/ngx_event_connect.c:24
#3  0x000000000043d5a8 in ngx_http_upstream_connect (r=0x54c3b30,
u=0x54ae250) at src/http/ngx_http_upstream.c:1103
#4  0x000000000043ee0a in ngx_http_upstream_init_request (r=0x54c3b30)
at src/http/ngx_http_upstream.c:631
#5  0x00000000004354a5 in ngx_http_read_client_request_body
(r=0x54c3b30, post_handler=0x43f310 <ngx_http_upstream_init>)
    at src/http/ngx_http_request_body.c:154
#6  0x00000000004572d6 in ngx_http_proxy_handler (r=0x54c3b30) at
src/http/modules/ngx_http_proxy_module.c:617
#7  0x000000000042b47c in ngx_http_core_content_phase (r=0x54c3b30,
ph=0x583ffd8) at src/http/ngx_http_core_module.c:1365
#8  0x0000000000426967 in ngx_http_core_run_phases (r=0x54c3b30) at
src/http/ngx_http_core_module.c:861
#9  0x000000000042fa66 in ngx_http_process_request (r=0x54c3b30) at
src/http/ngx_http_request.c:1665
#10 0x00000000004306a4 in ngx_http_process_request_line (rev=0x5843fc0)
at src/http/ngx_http_request.c:911
#11 0x0000000000419e86 in ngx_event_process_posted (cycle=<value
optimized out>, posted=0x68bd88) at src/event/ngx_event_posted.c:39
#12 0x000000000041f608 in ngx_worker_process_cycle (cycle=0x560f670,
data=<value optimized out>) at src/os/unix/ngx_process_cycle.c:801
#13 0x000000000041dd69 in ngx_spawn_process (cycle=0x560f670,
proc=0x41f550 <ngx_worker_process_cycle>, data=0x0, name=0x466581
"worker process", 
    respawn=-4) at src/os/unix/ngx_process.c:196
#14 0x000000000041ebeb in ngx_start_worker_processes (cycle=0x560f670,
n=8, type=-4) at src/os/unix/ngx_process_cycle.c:360
#15 0x000000000041ff88 in ngx_master_process_cycle (cycle=0x560f670) at
src/os/unix/ngx_process_cycle.c:249
#16 0x00000000004060d9 in main (argc=1, argv=<value optimized out>) at
src/core/nginx.c:405
(gdb) n
425                     rrp->current =
ngx_http_upstream_get_peer(rrp->peers);
(gdb) n
435                     if (!(rrp->tried[n] & m)) {
(gdb) n
460                     if (pc->tries == 0) {
(gdb) n
464                     if (--i == 0) {
(gdb) n
425                     rrp->current =
ngx_http_upstream_get_peer(rrp->peers);
(gdb) n
435                     if (!(rrp->tried[n] & m)) {
(gdb) n
460                     if (pc->tries == 0) {
(gdb) n
464                     if (--i == 0) {
(gdb) n
425                     rrp->current =
ngx_http_upstream_get_peer(rrp->peers);
(gdb) n
435                     if (!(rrp->tried[n] & m)) {
(gdb) n
460                     if (pc->tries == 0) {
(gdb) n
464                     if (--i == 0) {
(gdb) 





I found i made a mistake in nginx.conf, i include a virtual host
configuation like this:
        upstream test_servers {
                #server 10.0.7.4:80 ;
                server 10.0.7.5:80 backup;
                #server 10.0.7.6:80 ;
                #server 10.0.7.7:80 ;
        }

    server {
        listen       80;
        server_name  test.org ;

        access_log  /data1/logs/$host.access.log  main;

        location / {
                include proxy.conf;
                proxy_pass    http://test_servers;
        }


    }

there was only one server in upstream,which marked 'backup'. after some
test,i found this is the reason.

but when i test the nginx.conf syntax by using nginx -t,the result is
ok.

[root at host-22 ~]# /usr/local/nginx/sbin/nginx -t
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is
ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is
successful
[root at host-22 ~]#

i think nginx should warn me when that situation,haha...

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,218259,218315#msg-218315



More information about the nginx mailing list