nginx segfault around ~600 requests per second (reverse proxy)

[yomuppet]

Hello,

Sorry for the delayed reply, it has been taking longer to hit a segfault (which is good), but still hitting some. Here is some backtrace output:



(gdb) bt
#0  0x0000003967cd3443 in __epoll_wait_nocancel () from /lib64/libc.so.6
#1  0x000000000042890d in ngx_epoll_process_events (cycle=0xc1e1050, timer=, flags=1) at src/event/modules/ngx_epoll_module.c:530
#2  0x0000000000421f08 in ngx_process_events_and_timers (cycle=0xc1e1050) at src/event/ngx_event.c:245
#3  0x0000000000427740 in ngx_worker_process_cycle (cycle=0xc1e1050, data=) at src/os/unix/ngx_process_cycle.c:795
#4  0x0000000000426087 in ngx_spawn_process (cycle=0xc1e1050, proc=0x427678 , data=0x0, name=0x46f721 "worker process", respawn=2)
    at src/os/unix/ngx_process.c:196
#5  0x0000000000427f3a in ngx_master_process_cycle (cycle=0xc1e1050) at src/os/unix/ngx_process_cycle.c:612
#6  0x000000000040edfa in main (argc=22, argv=0xc1dff30) at src/core/nginx.c:399
(gdb) bt
#0  0x0000003967cd3443 in __epoll_wait_nocancel () from /lib64/libc.so.6
#1  0x000000000042890d in ngx_epoll_process_events (cycle=0xc1e1050, timer=, flags=1) at src/event/modules/ngx_epoll_module.c:530
#2  0x0000000000421f08 in ngx_process_events_and_timers (cycle=0xc1e1050) at src/event/ngx_event.c:245
#3  0x0000000000427740 in ngx_worker_process_cycle (cycle=0xc1e1050, data=) at src/os/unix/ngx_process_cycle.c:795
#4  0x0000000000426087 in ngx_spawn_process (cycle=0xc1e1050, proc=0x427678 , data=0x0, name=0x46f721 "worker process", respawn=2)
    at src/os/unix/ngx_process.c:196
#5  0x0000000000427f3a in ngx_master_process_cycle (cycle=0xc1e1050) at src/os/unix/ngx_process_cycle.c:612
#6  0x000000000040edfa in main (argc=22, argv=0xc1dff30) at src/core/nginx.c:399
(gdb) c
Continuing.
Program received signal SIGSEGV, Segmentation fault.
ngx_linux_sendfile_chain (c=0x2b31298aab70, in=0xc7bd768, limit=2147479551) at src/os/unix/ngx_linux_sendfile_chain.c:131
131                     iov->iov_len += (size_t) size;
(gdb) bt
#0  ngx_linux_sendfile_chain (c=0x2b31298aab70, in=0xc7bd768, limit=2147479551) at src/os/unix/ngx_linux_sendfile_chain.c:131
#1  0x000000000041186a in ngx_chain_writer (data=, in=) at src/core/ngx_output_chain.c:657
#2  0x0000000000412056 in ngx_output_chain (ctx=0xc7bd0b0, in=0x0) at src/core/ngx_output_chain.c:205
#3  0x000000000044540c in ngx_http_upstream_send_request (r=0xca55610, u=0xc7bd020) at src/http/ngx_http_upstream.c:1321
#4  0x0000000000445ef8 in ngx_http_upstream_send_request_handler (r=0x0, u=0x0) at src/http/ngx_http_upstream.c:1422
#5  0x000000000044399f in ngx_http_upstream_handler (ev=) at src/http/ngx_http_upstream.c:874
#6  0x00000000004220be in ngx_event_process_posted (cycle=, posted=0x0) at src/event/ngx_event_posted.c:39
#7  0x0000000000421f80 in ngx_process_events_and_timers (cycle=0xc1e1050) at src/event/ngx_event.c:272
#8  0x0000000000427740 in ngx_worker_process_cycle (cycle=0xc1e1050, data=) at src/os/unix/ngx_process_cycle.c:795
#9  0x0000000000426087 in ngx_spawn_process (cycle=0xc1e1050, proc=0x427678 , data=0x0, name=0x46f721 "worker process", respawn=2)
    at src/os/unix/ngx_process.c:196
#10 0x0000000000427f3a in ngx_master_process_cycle (cycle=0xc1e1050) at src/os/unix/ngx_process_cycle.c:612
#11 0x000000000040edfa in main (argc=22, argv=0xc1dff30) at src/core/nginx.c:399


(gdb) c
Continuing.
p iov
p prev
p cl->buf
p cl->buf->pos
up
detach

Program received signal SIGSEGV, Segmentation fault.
ngx_linux_sendfile_chain (c=0x2b21152bed40, in=0x48bfaf8, limit=2147479551) at src/os/unix/ngx_linux_sendfile_chain.c:131
131                     iov->iov_len += (size_t) size;
(gdb) p iov
$1 = (struct iovec *) 0x0
(gdb) p prev
$2 = (u_char *) 0x0
(gdb) p cl->buf
Cannot access memory at address 0x0
(gdb) p cl->buf->pos
Cannot access memory at address 0x0
(gdb) up
#1  0x000000000041186a in ngx_chain_writer (data=, in=) at src/core/ngx_output_chain.c:657
657         ctx->out = c->send_chain(c, ctx->out, ctx->limit);
(gdb) detach
LND: Sending signal 11 to Thread 0x2b2114c6b750 (LWP 21815)
Detaching from program: /opt/nginx/sbin/nginx, process 21815




If anyone has further insight, would be much appreciated. Thanks.

J

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,45615,59278#msg-59278