Nginx crash - possibly keepalives

[Matthieu Tourne]

On Wed, Oct 19, 2011 at 2:24 AM, Maxim Dounin <mdounin at mdounin.ru> wrote:

> Hello!
>
> On Tue, Oct 18, 2011 at 06:32:53PM -0700, Matthieu Tourne wrote:
>
> > So in order to try to reproduce the problem locally, I made this debug
> > module :
> > https://github.com/mtourne/ngx_massive_chunker
> >
> > This is meant to have 2 nginx chained one to the other.
> > nginx#1 has  proxy_pass to nginx#2 while using keepalives
> > nginx#2 has mass_chunk on, which for this extreme test will chunk a 15M
> file
> > into 10 bytes chunk (in different tcp packets or not).
> >
> > I wasn't able to reproduce the segfault, but when I'm using
> > proxy_buffering off it works fine.
>
> To reproduce the segfault you have to:
>
> 1. Fill client socket buffers to make sure no buffers will be
> freed at (2), i.e. all buffers will become busy.  This should be
> done carefully to avoid making some proxy buffers busy here, as
> this is likely to make (2) impossible.
>
> 2. Trigger "all buffers in the out chain" problem by sending multiple
> small chunks at once.  Amount of data must be less than
> proxy_busy_buffers, but total input size (including chunked encoding
> overhead) must be big enough to consume all proxy_buffers
> (including first one, proxy_buffer_size).
>
> 3. Send some more data to actually trigger segfault due to all
> buffers being busy.
>
> I'm able to reproduce it here (using specially crafted backend
> server and specially crafted client), and I'm not really intrested
> in another artificial reproduction.
>
> > But, when I'm using proxy_buffering and proxy_caching (with sendfile on,
> and
> > directio 2M).
> > It seems like readv() returns 0, but finalize_request returns NGX_AGAIN.
>
> ENOPARSE
>
> The ngx_http_finalize_request() is void, it doesn't return anything.
>
> You may want to provide debug log to make it clear what are you
> talking about.
>

I'm not exactly sure what part of it is relevant, so instead of posting
pages of log I provided steps to reproduce a configuration where nginx shows
an erratic behavior, by not outputing the entire response while doing a
proxy_pass.

I'll be experimenting more, and I'll let you know if I find the origin of
this issue.

M.


>
> Maxim Dounin
>
> > I'm not sure that test is relevant, and what I'm really benching here.
> And
> > if it might not be the tcp layer directly?
> > But maybe the failure should be explicit.
> >
> > The behavior was the same with or without the patch. I've also tried
> various
> > combination of proxy buffers, sendfile, directio.
> >
> > Any thoughts ?
> >
> > Thank you!
> > Matthieu.
> >
> > On Fri, Oct 14, 2011 at 12:43 PM, Maxim Dounin <mdounin at mdounin.ru>
> wrote:
> >
> > > Hello!
> > >
> > > On Wed, Oct 12, 2011 at 09:22:41PM +0400, Maxim Dounin wrote:
> > >
> > > > Hello!
> > > >
> > > > On Wed, Oct 12, 2011 at 09:32:41AM -0700, Matthieu Tourne wrote:
> > > >
> > > > > On Wed, Oct 12, 2011 at 5:20 AM, Maxim Dounin <mdounin at mdounin.ru>
> > > wrote:
> > > > >
> > > > > > Hello!
> > > > > >
> > > > > > On Tue, Oct 11, 2011 at 07:53:36PM -0700, Matthieu Tourne wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > After turning on keepalives, we've been seeing one crash pretty
> > > > > > > consistently.
> > > > > > > We're running nginx 1.1.5 and here is the backtrace :
> > >
> > > [...]
> > >
> > > > Ok, it looks like I'm right and all buffers are in busy chain.
> > > > Likely this happens due to upstream sending response in many small
> > > > chunks.
> > > >
> > > > I'll try to reproduce it here and provide proper fix.
> > >
> > > Please try the attached patch.
> > >
> > > Maxim Dounin
> > >
> > > _______________________________________________
> > > nginx mailing list
> > > nginx at nginx.org
> > > http://mailman.nginx.org/mailman/listinfo/nginx
> > >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20111019/26c10e19/attachment.html>