nginx & core dump

ak84 nginx-forum at nginx.us
Sun Sep 23 20:03:50 UTC 2012


Добрый вечер.



Есть сервер на FreeBSD 8.2-RELEASE amd64, nginx выступает в роли
frontend-сервера, для нескольких доменов, проксирующего запросы к
backend-серверам через fastcgi и proxy_pass
При включении debug-лога:

2012/09/23 16:32:46 [debug] 16358#0: *369779 close http connection: 257
2012/09/23 16:32:46 [debug] 16358#0: *369779 free: 000000080230B000
2012/09/23 16:32:46 [debug] 16358#0: *369779 free: 0000000801839800
2012/09/23 16:32:46 [debug] 16358#0: *369779 free: 0000000801861400
2012/09/23 16:32:46 [debug] 16358#0: *369779 free: 0000000802333A00, unused:
8
2012/09/23 16:32:46 [debug] 16358#0: *369779 free: 0000000802333D00, unused:
0
2012/09/23 16:32:46 [debug] 16358#0: *369779 free: 0000000802333E00, unused:
208
2012/09/23 16:32:46 [debug] 16358#0: kevent: 258: ft:-2 fl:0020 ff:00000000
d:34080 ud:00000008030E7E90
2012/09/23 16:32:46 [debug] 16358#0: *369701 http run request:
"/images/icon.gif?1348318282"
2012/09/23 16:32:46 [debug] 16358#0: *369701 http upstream check client,
write event:1, "/images/icon.gif"
2012/09/23 16:32:46 [debug] 16358#0: timer delta: 3
2012/09/23 16:32:46 [debug] 16358#0: posted events 0000000000000000
2012/09/23 16:32:46 [debug] 16358#0: worker cycle
2012/09/23 16:32:46 [debug] 16358#0: accept mutex lock failed: 0
2012/09/23 16:32:46 [debug] 16358#0: kevent timer: 162, changes: 0
2012/09/23 16:32:46 [debug] 16358#0: kevent events: 1
2012/09/23 16:32:46 [debug] 16358#0: kevent: 259: ft:-1 fl:8020 ff:00000000
d:0 ud:0000000803006371
2012/09/23 16:32:46 [debug] 16358#0: *369727 http run request:
"/?c=chat&m=save_reply&owner_id=3699228&selected_tab=2&inbox=1&to=1634790"
2012/09/23 16:32:46 [debug] 16358#0: *369727 http read client request body
2012/09/23 16:32:46 [debug] 16358#0: *369727 add cleanup: 00000008031DCA90
2012/09/23 16:32:46 [debug] 16358#0: *369727 hashed path:
/var/tmp/nginx/client_body_temp/0000001175
2012/09/23 16:32:46 [debug] 16358#0: *369727 temp fd:257
2012/09/23 16:32:46 [warn] 16358#0: *369727 a client request body is
buffered to a temporary file /var/tmp/nginx/client_body_temp/0000001175
while sending request to upst
ream, client: client_ip, server: www.test.com, request: "POST
/?id=user&owner_id=1&selected_id=2&inbox=1 HTTP/1.1", upstream: "ht
tp://backend_ip/?id=user&owner_id=1&selected_id=2&inbox=1", host:
"test.com", referrer:
"http://test.com/?id=user&owner_id=1&selected_id=2&inbox=1"
2012/09/23 16:32:46 [debug] 16358#0: *369727 write: 257, 00000008022ADEE8,
153, 0
2012/09/23 16:32:46 [debug] 16358#0: *369727 recv: eof:1, avail:0, err:0
2012/09/23 16:32:46 [debug] 16358#0: *369727 http client request body recv
0
2012/09/23 16:32:46 [info] 16358#0: *369727 client closed prematurely
connection while sending request to upstream, client: client_ip, server:
www.test.com, re
quest: "POST /?id=user&owner_id=1&selected_id=2&inbox=1 HTTP/1.1", upstream:
"http://backend_ip/?id=user&owner_id=1&selected_id=2&inbox=1", host:
"test.com", referrer:
"http://test.com/?id=user&owner_id=1&selected_id=2&inbox=1"
2012/09/23 16:32:46 [debug] 16358#0: *369727 http finalize request: 400,
"/?id=user&owner_id=1&selected_id=2&inbox=1" a:1, c:1
2012/09/23 16:32:46 [debug] 16358#0: *369727 http terminate request count:1
2012/09/23 16:32:46 [debug] 16358#0: *369727 cleanup http upstream request:
"/"
2012/09/23 16:32:46 [debug] 16358#0: *369727 finalize http upstream request:
-4
2012/09/23 16:32:46 [debug] 16358#0: *369727 finalize http proxy request
2012/09/23 16:32:46 [debug] 16358#0: *369727 free rr peer 2 0
2012/09/23 16:32:46 [debug] 16358#0: *369727 close http upstream connection:
267
2012/09/23 16:32:46 [debug] 16358#0: *369727 event timer del: 267:
1348403626758
2012/09/23 16:32:46 [debug] 16358#0: *369727 http finalize request: -4,
"/?id=user&owner_id=1&selected_id=2&inbox=1" a:1, c:1
2012/09/23 16:32:46 [debug] 16358#0: *369727 set http keepalive handler
2012/09/23 16:32:46 [debug] 16358#0: *369727 http close request
2012/09/23 16:32:46 [debug] 16358#0: *369727 http log handler
2012/09/23 16:32:46 [debug] 16358#0: *369727 run cleanup: 00000008031DCA90
2012/09/23 16:32:46 [debug] 16358#0: *369727 file cleanup: fd:257
2012/09/23 16:32:46 [debug] 16358#0: *369727 free: 00000008022AD000, unused:
3
2012/09/23 16:32:46 [debug] 16358#0: *369727 free: 00000008031DC000, unused:
851
2012/09/23 16:32:46 [debug] 16358#0: *369727 event timer add: 259:
65000:1348403631780
2012/09/23 16:32:46 [debug] 16358#0: *369727 free: 000000080185F600
2012/09/23 16:32:46 [debug] 16358#0: *369727 free: 0000000801838000
2012/09/23 16:32:46 [debug] 16358#0: *369727 hc free: 0000000000000000 0
2012/09/23 16:32:46 [debug] 16358#0: *369727 hc busy: 00000008022CAD90 1
2012/09/23 16:32:46 [debug] 16358#0: *369727 free: 00000008019FA000
2012/09/23 16:32:46 [debug] 16358#0: *369727 tcp_nodelay
2012/09/23 16:32:46 [alert] 16352#0: worker process 16358 exited on signal
10 (core dumped)

nginx version: nginx/0.8.54
TLS SNI support enabled
configure arguments: --prefix=/usr/local/etc/nginx --with-cc-opt='-I
/usr/local/include' --with-ld-opt='-L /usr/local/lib'
--conf-path=/usr/local/etc/nginx/nginx.conf
--sbin-path=/usr/local/sbin/nginx --pid-path=/var/run/nginx.pid
--error-log-path=/var/log/nginx-error.log --user=www --group=www
--with-debug --http-client-body-temp-path=/var/tmp/nginx/client_body_temp
--http-fastcgi-temp-path=/var/tmp/nginx/fastcgi_temp
--http-proxy-temp-path=/var/tmp/nginx/proxy_temp
--http-scgi-temp-path=/var/tmp/nginx/scgi_temp
--http-uwsgi-temp-path=/var/tmp/nginx/uwsgi_temp
--http-log-path=/var/log/nginx-access.log --without-http-cache
--add-module=/usr/ports/www/nginx/work/ngx_headers_more_module-0.13
--with-http_image_filter_module --with-http_ssl_module
--with-http_stub_status_module --with-pcre

В логах стало появляться:
pid 19220 (nginx), uid 65534: exited on signal 10 (core dumped)

Смотри core-файл:

#0  0x000000000044ec03 in ngx_http_terminate_request (r=0x801862300, rc=400)
at src/http/ngx_http_request.c:2069
2069	src/http/ngx_http_request.c: No such file or directory.
	in src/http/ngx_http_request.c
(gdb) bt
#0  0x000000000044ec03 in ngx_http_terminate_request (r=0x801862300, rc=400)
at src/http/ngx_http_request.c:2069
#1  0x000000000044e606 in ngx_http_finalize_request (r=0x801862300, rc=400)
at src/http/ngx_http_request.c:1903
#2  0x000000000045900b in ngx_http_read_client_request_body_handler
(r=0x801862300) at src/http/ngx_http_request_body.c:254
#3  0x000000000044e28e in ngx_http_request_handler (ev=0x803011120) at
src/http/ngx_http_request.c:1791
#4  0x0000000000429b84 in ngx_event_process_posted (cycle=0x801842050,
posted=0x5cfb20) at src/event/ngx_event_posted.c:39
#5  0x00000000004277dd in ngx_process_events_and_timers (cycle=0x801842050)
at src/event/ngx_event.c:272
#6  0x0000000000433f03 in ngx_worker_process_cycle (cycle=0x801842050,
data=0x0) at src/os/unix/ngx_process_cycle.c:795
#7  0x00000000004310bd in ngx_spawn_process (cycle=0x801842050,
proc=0x433d2e <ngx_worker_process_cycle>, data=0x0, name=0x4a7271 "worker
process", respawn=6)
    at src/os/unix/ngx_process.c:196
#8  0x0000000000433987 in ngx_reap_children (cycle=0x801842050) at
src/os/unix/ngx_process_cycle.c:612
#9  0x0000000000432748 in ngx_master_process_cycle (cycle=0x801842050) at
src/os/unix/ngx_process_cycle.c:180
#10 0x0000000000405c81 in main (argc=1, argv=0x7fffffffed68) at
src/core/nginx.c:401
(gdb) backtrace
#0  0x000000000044ec03 in ngx_http_terminate_request (r=0x801862300, rc=400)
at src/http/ngx_http_request.c:2069
#1  0x000000000044e606 in ngx_http_finalize_request (r=0x801862300, rc=400)
at src/http/ngx_http_request.c:1903
#2  0x000000000045900b in ngx_http_read_client_request_body_handler
(r=0x801862300) at src/http/ngx_http_request_body.c:254
#3  0x000000000044e28e in ngx_http_request_handler (ev=0x803011120) at
src/http/ngx_http_request.c:1791
#4  0x0000000000429b84 in ngx_event_process_posted (cycle=0x801842050,
posted=0x5cfb20) at src/event/ngx_event_posted.c:39
#5  0x00000000004277dd in ngx_process_events_and_timers (cycle=0x801842050)
at src/event/ngx_event.c:272
#6  0x0000000000433f03 in ngx_worker_process_cycle (cycle=0x801842050,
data=0x0) at src/os/unix/ngx_process_cycle.c:795
#7  0x00000000004310bd in ngx_spawn_process (cycle=0x801842050,
proc=0x433d2e <ngx_worker_process_cycle>, data=0x0, name=0x4a7271 "worker
process", respawn=6)
    at src/os/unix/ngx_process.c:196
#8  0x0000000000433987 in ngx_reap_children (cycle=0x801842050) at
src/os/unix/ngx_process_cycle.c:612
#9  0x0000000000432748 in ngx_master_process_cycle (cycle=0x801842050) at
src/os/unix/ngx_process_cycle.c:180
#10 0x0000000000405c81 in main (argc=1, argv=0x7fffffffed68) at
src/core/nginx.c:401
(gdb) backtrace full
#0  0x000000000044ec03 in ngx_http_terminate_request (r=0x801862300, rc=400)
at src/http/ngx_http_request.c:2069
	cln = (ngx_http_cleanup_t *) 0x5a5a5a5a5a5a5a5a
	mr = (ngx_http_request_t *) 0x801862300
	e = (ngx_http_ephemeral_t *) 0x801ee1218
#1  0x000000000044e606 in ngx_http_finalize_request (r=0x801862300, rc=400)
at src/http/ngx_http_request.c:1903
	c = (ngx_connection_t *) 0x801c19790
	pr = (ngx_http_request_t *) 0x8031cbe98
	clcf = (ngx_http_core_loc_conf_t *) 0x803010240
#2  0x000000000045900b in ngx_http_read_client_request_body_handler
(r=0x801862300) at src/http/ngx_http_request_body.c:254
	rc = 400
#3  0x000000000044e28e in ngx_http_request_handler (ev=0x803011120) at
src/http/ngx_http_request.c:1791
	c = (ngx_connection_t *) 0x801c19790
	r = (ngx_http_request_t *) 0x801862300
	ctx = (ngx_http_log_ctx_t *) 0x801d85ea0
#4  0x0000000000429b84 in ngx_event_process_posted (cycle=0x801842050,
posted=0x5cfb20) at src/event/ngx_event_posted.c:39
	ev = (ngx_event_t *) 0x803011120
#5  0x00000000004277dd in ngx_process_events_and_timers (cycle=0x801842050)
at src/event/ngx_event.c:272
	flags = 3
	timer = 505
	delta = 0
#6  0x0000000000433f03 in ngx_worker_process_cycle (cycle=0x801842050,
data=0x0) at src/os/unix/ngx_process_cycle.c:795
	i = 1
	c = (ngx_connection_t *) 0x7fffffffe9a0
#7  0x00000000004310bd in ngx_spawn_process (cycle=0x801842050,
proc=0x433d2e <ngx_worker_process_cycle>, data=0x0, name=0x4a7271 "worker
process", respawn=6)
    at src/os/unix/ngx_process.c:196
	on = 1
	pid = 0
	s = 6
#8  0x0000000000433987 in ngx_reap_children (cycle=0x801842050) at
src/os/unix/ngx_process_cycle.c:612
	i = 6
	n = 8
	live = 1
	ch = {command = 2, pid = 19222, slot = 6, fd = -1}
	ccf = (ngx_core_conf_t *) 0x505f560f
#9  0x0000000000432748 in ngx_master_process_cycle (cycle=0x801842050) at
src/os/unix/ngx_process_cycle.c:180
	title = 0x8023c504c "master process /usr/local/sbin/nginx"
	p = (u_char *) 0x8023c5070 ""
	size = 37
	i = 1
	n = 6
	sigio = 0
	set = {__bits = {0, 0, 0, 0}}
	itv = {it_interval = {tv_sec = 0, tv_usec = 0}, it_value = {tv_sec = 0,
tv_usec = 0}}
	live = 1
	delay = 0
	ls = (ngx_listening_t *) 0x6
	ccf = (ngx_core_conf_t *) 0x801843008
#10 0x0000000000405c81 in main (argc=1, argv=0x7fffffffed68) at
src/core/nginx.c:401
---Type <return> to continue, or q <return> to quit---
	i = 48
	log = (ngx_log_t *) 0x5cd800
	cycle = (ngx_cycle_t *) 0x801842050
	init_cycle = {conf_ctx = 0x0, pool = 0x801836800, log = 0x5cd800, new_log =
{log_level = 0, file = 0x0, connection = 0, handler = 0, data = 0x0, action
= 0x0}, 
  files = 0x0, free_connections = 0x0, free_connection_n = 0, listening =
{elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, pathes = {elts =
0x0, nelts = 0, 
    size = 0, nalloc = 0, pool = 0x0}, open_files = {last = 0x0, part =
{elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0},
shared_memory = {
    last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc
= 0, pool = 0x0}, connection_n = 0, files_n = 0, connections = 0x0,
read_events = 0x0, 
  write_events = 0x0, old_cycle = 0x0, conf_file = {len = 31, data =
0x4a36c8 "/usr/local/etc/nginx/nginx.conf"}, conf_param = {len = 0, data =
0x0}, conf_prefix = {
    len = 21, data = 0x4a36c8 "/usr/local/etc/nginx/nginx.conf"}, prefix =
{len = 21, data = 0x4a36b0 "/usr/local/etc/nginx/"}, lock_file = {len = 0,
data = 0x0}, 
  hostname = {len = 0, data = 0x0}}
	ccf = (ngx_core_conf_t *) 0x801843008

Posted at Nginx Forum: http://forum.nginx.org/read.php?21,230962,230962#msg-230962



Подробная информация о списке рассылки nginx-ru