nginx worker process hang,cpu load 100%
Long Wan
nginx-forum at nginx.us
Mon Nov 14 14:37:29 UTC 2011
Hello,Maxim. Thanks for you help.
I reproduce the problem in nginx-1.0.9,
[root at host-22 ~]# /usr/local/nginx/sbin/nginx -V
nginx: nginx version: nginx/1.0.9
nginx: built by gcc 4.1.2 20080704 (Red Hat 4.1.2-51)
nginx: TLS SNI support disabled
nginx: configure arguments: --user=www --group=www
--prefix=/usr/local/nginx --with-http_stub_status_module
--with-http_ssl_module --with-openssl-opt=enable-tlsext
--with-http_sub_module --with-cc-opt=-O2 --with-cpu-opt=opteron
--add-module=../ngx_cache_purge-1.4
[root at host-22 ~]#
[root at host-22 ~]#
[root at host-22 ~]# ps aux|grep -e CPU -e nginx
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME
COMMAND
root 569 0.0 0.0 49016 6884 ? Ss 21:59 0:00 nginx:
master process /usr/local/nginx/sbin/nginx
www 587 97.3 0.0 49380 7556 ? R 22:00 10:16 nginx:
worker process
www 588 93.8 0.0 49380 7556 ? R 22:00 9:54 nginx:
worker process
www 614 43.1 0.0 49412 7584 ? T 22:01 4:07 nginx:
worker process
root 781 0.0 0.1 95984 19684 pts/0 S+ 22:06 0:00 gdb
/usr/local/nginx/sbin/nginx 614
www 876 0.0 0.0 50504 8464 ? S 22:10 0:00 nginx:
worker process
www 877 0.0 0.0 50504 8464 ? S 22:10 0:00 nginx:
worker process
www 878 0.0 0.0 50504 8464 ? S 22:10 0:00 nginx:
worker process
www 879 0.0 0.0 50504 8660 ? S 22:10 0:00 nginx:
worker process
www 880 0.0 0.0 50504 8464 ? S 22:10 0:00 nginx:
worker process
www 881 0.0 0.0 50504 8464 ? S 22:10 0:00 nginx:
worker process
www 882 0.0 0.0 50504 8464 ? S 22:10 0:00 nginx:
worker process
www 883 0.0 0.0 50504 8464 ? S 22:10 0:00 nginx:
worker process
root 954 0.0 0.0 61168 788 pts/1 S+ 22:10 0:00 grep -e
CPU -e nginx
[root at host-22 ~]#
[root at host-22 ~]# gdb /usr/local/nginx/sbin/nginx 614
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show
copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/nginx/sbin/nginx...done.
Attaching to program: /usr/local/nginx/sbin/nginx, process 614
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libpcre.so.0...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libpcre.so.0
Reading symbols from /lib64/libssl.so.6...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libssl.so.6
Reading symbols from /lib64/libcrypto.so.6...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libcrypto.so.6
Reading symbols from /lib64/libdl.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /usr/lib64/libz.so.1...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libz.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/libgssapi_krb5.so.2...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libkrb5.so.3...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /usr/lib64/libk5crypto.so.3...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libk5crypto.so.3
Reading symbols from /usr/lib64/libkrb5support.so.0...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libsepol.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libsepol.so.1
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libnss_files.so.2
ngx_http_upstream_get_peer (pc=0x54ae260, data=<value optimized out>) at
src/http/ngx_http_upstream_round_robin.c:632
632 if (reset++) {
(gdb) bt
#0 ngx_http_upstream_get_peer (pc=0x54ae260, data=<value optimized
out>) at src/http/ngx_http_upstream_round_robin.c:632
#1 ngx_http_upstream_get_round_robin_peer (pc=0x54ae260, data=<value
optimized out>) at src/http/ngx_http_upstream_round_robin.c:425
#2 0x000000000041a99c in ngx_event_connect_peer (pc=0x54ae960) at
src/event/ngx_event_connect.c:24
#3 0x000000000043d5a8 in ngx_http_upstream_connect (r=0x54c3b30,
u=0x54ae250) at src/http/ngx_http_upstream.c:1103
#4 0x000000000043ee0a in ngx_http_upstream_init_request (r=0x54c3b30)
at src/http/ngx_http_upstream.c:631
#5 0x00000000004354a5 in ngx_http_read_client_request_body
(r=0x54c3b30, post_handler=0x43f310 <ngx_http_upstream_init>)
at src/http/ngx_http_request_body.c:154
#6 0x00000000004572d6 in ngx_http_proxy_handler (r=0x54c3b30) at
src/http/modules/ngx_http_proxy_module.c:617
#7 0x000000000042b47c in ngx_http_core_content_phase (r=0x54c3b30,
ph=0x583ffd8) at src/http/ngx_http_core_module.c:1365
#8 0x0000000000426967 in ngx_http_core_run_phases (r=0x54c3b30) at
src/http/ngx_http_core_module.c:861
#9 0x000000000042fa66 in ngx_http_process_request (r=0x54c3b30) at
src/http/ngx_http_request.c:1665
#10 0x00000000004306a4 in ngx_http_process_request_line (rev=0x5843fc0)
at src/http/ngx_http_request.c:911
#11 0x0000000000419e86 in ngx_event_process_posted (cycle=<value
optimized out>, posted=0x68bd88) at src/event/ngx_event_posted.c:39
#12 0x000000000041f608 in ngx_worker_process_cycle (cycle=0x560f670,
data=<value optimized out>) at src/os/unix/ngx_process_cycle.c:801
#13 0x000000000041dd69 in ngx_spawn_process (cycle=0x560f670,
proc=0x41f550 <ngx_worker_process_cycle>, data=0x0, name=0x466581
"worker process",
respawn=-4) at src/os/unix/ngx_process.c:196
#14 0x000000000041ebeb in ngx_start_worker_processes (cycle=0x560f670,
n=8, type=-4) at src/os/unix/ngx_process_cycle.c:360
#15 0x000000000041ff88 in ngx_master_process_cycle (cycle=0x560f670) at
src/os/unix/ngx_process_cycle.c:249
#16 0x00000000004060d9 in main (argc=1, argv=<value optimized out>) at
src/core/nginx.c:405
(gdb) n
425 rrp->current =
ngx_http_upstream_get_peer(rrp->peers);
(gdb) n
435 if (!(rrp->tried[n] & m)) {
(gdb) n
460 if (pc->tries == 0) {
(gdb) n
464 if (--i == 0) {
(gdb) n
425 rrp->current =
ngx_http_upstream_get_peer(rrp->peers);
(gdb) n
435 if (!(rrp->tried[n] & m)) {
(gdb) n
460 if (pc->tries == 0) {
(gdb) n
464 if (--i == 0) {
(gdb) n
425 rrp->current =
ngx_http_upstream_get_peer(rrp->peers);
(gdb) n
435 if (!(rrp->tried[n] & m)) {
(gdb) n
460 if (pc->tries == 0) {
(gdb) n
464 if (--i == 0) {
(gdb)
I found i made a mistake in nginx.conf, i include a virtual host
configuation like this:
upstream test_servers {
#server 10.0.7.4:80 ;
server 10.0.7.5:80 backup;
#server 10.0.7.6:80 ;
#server 10.0.7.7:80 ;
}
server {
listen 80;
server_name test.org ;
access_log /data1/logs/$host.access.log main;
location / {
include proxy.conf;
proxy_pass http://test_servers;
}
}
there was only one server in upstream,which marked 'backup'. after some
test,i found this is the reason.
but when i test the nginx.conf syntax by using nginx -t,the result is
ok.
[root at host-22 ~]# /usr/local/nginx/sbin/nginx -t
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is
ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is
successful
[root at host-22 ~]#
i think nginx should warn me when that situation,haha...
Posted at Nginx Forum: http://forum.nginx.org/read.php?2,218259,218315#msg-218315
More information about the nginx
mailing list