Nginx crash - possibly keepalives

[Maxim Dounin]

Hello!

On Tue, Oct 11, 2011 at 07:53:36PM -0700, Matthieu Tourne wrote:

> Hi all,
> 
> After turning on keepalives, we've been seeing one crash pretty
> consistently.
> We're running nginx 1.1.5 and here is the backtrace :
> 
> (gdb) bt
> #0  ngx_write_chain_to_file (file=0x173dbd0, cl=0x0, offset=26779,
> pool=0x162b850) at src/os/unix/ngx_files.c:162
> #1  0x000000000040ea3e in ngx_write_chain_to_temp_file (tf=0x173dbd0,
> chain=0x0) at src/core/ngx_file.c:35
> #2  0x000000000041c534 in ngx_event_pipe_write_chain_to_temp_file
> (p=0x173d048) at src/event/ngx_event_pipe.c:771

Do you have any local patches here or it's just some debug logging 
added?

$ grep -n ngx_write_chain_to_temp_file src/event/ngx_event_pipe.c
521:                if (ngx_write_chain_to_temp_file(p->temp_file, &file)
765:    if (ngx_write_chain_to_temp_file(p->temp_file, out) == NGX_ERROR) {

> #3  0x000000000041cd3d in ngx_event_pipe_read_upstream (p=0x173d048,
> do_write=0) at src/event/ngx_event_pipe.c:234
> #4  ngx_event_pipe (p=0x173d048, do_write=0) at
> src/event/ngx_event_pipe.c:50
> #5  0x000000000043e1b6 in ngx_http_upstream_process_upstream (r=0x1ae4bb0,
> u=0x1afb4f8) at src/http/ngx_http_upstream.c:2704
> #6  0x000000000043e612 in ngx_http_upstream_handler (ev=0x689b) at
> src/http/ngx_http_upstream.c:936
> #7  0x000000000041b1f6 in ngx_event_process_posted (cycle=<value optimized
> out>, posted=0x0) at src/event/ngx_event_posted.c:39
> #8  0x000000000041b0c6 in ngx_process_events_and_timers (cycle=0x151bbf0) at
> src/event/ngx_event.c:272
> #9  0x0000000000420bc3 in ngx_worker_process_cycle (cycle=0x151bbf0,
> data=<value optimized out>) at src/os/unix/ngx_process_cycle.c:801
> #10 0x000000000041f4cc in ngx_spawn_process (cycle=0x151bbf0, proc=0x420aeb
> <ngx_worker_process_cycle>, data=<value optimized out>, name=0x470dc9
> "worker process", respawn=5) at src/os/unix/ngx_process.c:196
> #11 0x0000000000421359 in ngx_reap_children (cycle=0x151bbf0) at
> src/os/unix/ngx_process_cycle.c:617
> #12 ngx_master_process_cycle (cycle=0x151bbf0) at
> src/os/unix/ngx_process_cycle.c:180
> #13 0x000000000040700a in main (argc=<value optimized out>, argv=<value
> optimized out>) at src/core/nginx.c:405
> 
> It seems that p->in is NULL in ngx_event_pipe_write_chain_to_temp_file() but
> I can't figure out why.
> It's using the proxy module as an upstream.
> 
> Let me know I could provide more info to help!

Could you please show:

fr 2
p *p
p p->bufs
p *p->temp_file

fr 5
p *r
p *u
p u->headers_in

I think I see what's going on here, just to confirm.  The 
underlying issue seems to be one identified in commit message 
here:

http://trac.nginx.org/nginx/changeset/4136/nginx

(all available buffers in the p->out chain)

Setting proxy_busy_buffers to something lower compared to 
proxy_buffers may help as a workaround.

Maxim Dounin