Nginx crash - possibly keepalives

[Maxim Dounin]

Hello!

On Wed, Oct 19, 2011 at 10:49:26AM -0700, Matthieu Tourne wrote:

> On Wed, Oct 19, 2011 at 2:24 AM, Maxim Dounin <mdounin at mdounin.ru> wrote:
> 
> > Hello!
> >
> > On Tue, Oct 18, 2011 at 06:32:53PM -0700, Matthieu Tourne wrote:

[...]

> > > But, when I'm using proxy_buffering and proxy_caching (with sendfile on,
> > and
> > > directio 2M).
> > > It seems like readv() returns 0, but finalize_request returns NGX_AGAIN.
> >
> > ENOPARSE
> >
> > The ngx_http_finalize_request() is void, it doesn't return anything.
> >
> > You may want to provide debug log to make it clear what are you
> > talking about.
> >
> 
> I'm not exactly sure what part of it is relevant, so instead of posting
> pages of log I provided steps to reproduce a configuration where nginx shows
> an erratic behavior, by not outputing the entire response while doing a
> proxy_pass.

So the issue you are seeing is truncated response, right?

I'm not able to reproduce it here (though I've tested under 
FreeBSD, not Linux).  Do you see sendfile and/or directio have any 
influence?

And the point still remains: please provide debug log.  Obviously 
posting it here is a bad idea as it's likely to be huge, though 
making gzipped version available for download would be really 
helpfull.

> I'll be experimenting more, and I'll let you know if I find the origin of
> this issue.

Please note that testing with your module may be problematic, 
as there are at least the following issues with it:

1. "mass_chunk_multi_packets on" expected to cause truncated 
responses as it aborts response on != NGX_OK from next body 
filter (e.g. on NGX_AGAIN if network isn't as fast as reading from
disk for some reason).

2. checksums will not match as buffers from previous filters are 
marked as free before actual data are sent, thus causing mostly 
arbitrary data to be sent (again, if network isn't as fast as 
reading from disk for some reason).

Maxim Dounin

> 
> 
> M.
> 
> 
> >
> > Maxim Dounin
> >
> > > I'm not sure that test is relevant, and what I'm really benching here.
> > And
> > > if it might not be the tcp layer directly?
> > > But maybe the failure should be explicit.
> > >
> > > The behavior was the same with or without the patch. I've also tried
> > various
> > > combination of proxy buffers, sendfile, directio.
> > >
> > > Any thoughts ?
> > >
> > > Thank you!
> > > Matthieu.
> > >
> > > On Fri, Oct 14, 2011 at 12:43 PM, Maxim Dounin <mdounin at mdounin.ru>
> > wrote:
> > >
> > > > Hello!
> > > >
> > > > On Wed, Oct 12, 2011 at 09:22:41PM +0400, Maxim Dounin wrote:
> > > >
> > > > > Hello!
> > > > >
> > > > > On Wed, Oct 12, 2011 at 09:32:41AM -0700, Matthieu Tourne wrote:
> > > > >
> > > > > > On Wed, Oct 12, 2011 at 5:20 AM, Maxim Dounin <mdounin at mdounin.ru>
> > > > wrote:
> > > > > >
> > > > > > > Hello!
> > > > > > >
> > > > > > > On Tue, Oct 11, 2011 at 07:53:36PM -0700, Matthieu Tourne wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > After turning on keepalives, we've been seeing one crash pretty
> > > > > > > > consistently.
> > > > > > > > We're running nginx 1.1.5 and here is the backtrace :
> > > >
> > > > [...]
> > > >
> > > > > Ok, it looks like I'm right and all buffers are in busy chain.
> > > > > Likely this happens due to upstream sending response in many small
> > > > > chunks.
> > > > >
> > > > > I'll try to reproduce it here and provide proper fix.
> > > >
> > > > Please try the attached patch.
> > > >
> > > > Maxim Dounin
> > > >
> > > > _______________________________________________
> > > > nginx mailing list
> > > > nginx at nginx.org
> > > > http://mailman.nginx.org/mailman/listinfo/nginx
> > > >
> >