<br><br><div class="gmail_quote">On Wed, Oct 19, 2011 at 2:24 AM, Maxim Dounin <span dir="ltr"><<a href="mailto:mdounin@mdounin.ru">mdounin@mdounin.ru</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Hello!<br>
<div class="im"><br>
On Tue, Oct 18, 2011 at 06:32:53PM -0700, Matthieu Tourne wrote:<br>
<br>
> So in order to try to reproduce the problem locally, I made this debug<br>
> module :<br>
> <a href="https://github.com/mtourne/ngx_massive_chunker" target="_blank">https://github.com/mtourne/ngx_massive_chunker</a><br>
><br>
> This is meant to have 2 nginx chained one to the other.<br>
> nginx#1 has šproxy_pass to nginx#2 while using keepalives<br>
> nginx#2 has mass_chunk on, which for this extreme test will chunk a 15M file<br>
> into 10 bytes chunk (in different tcp packets or not).<br>
><br>
> I wasn't able to reproduce the segfault, but when I'm using<br>
> proxy_buffering off it works fine.<br>
<br>
</div>To reproduce the segfault you have to:<br>
<br>
1. Fill client socket buffers to make sure no buffers will be<br>
freed at (2), i.e. all buffers will become busy. šThis should be<br>
done carefully to avoid making some proxy buffers busy here, as<br>
this is likely to make (2) impossible.<br>
<br>
2. Trigger "all buffers in the out chain" problem by sending multiple<br>
small chunks at once. šAmount of data must be less than<br>
proxy_busy_buffers, but total input size (including chunked encoding<br>
overhead) must be big enough to consume all proxy_buffers<br>
(including first one, proxy_buffer_size).<br>
<br>
3. Send some more data to actually trigger segfault due to all<br>
buffers being busy.<br>
<br>
I'm able to reproduce it here (using specially crafted backend<br>
server and specially crafted client), and I'm not really intrested<br>
in another artificial reproduction.<br>
<div class="im"><br>
> But, when I'm using proxy_buffering and proxy_caching (with sendfile on, and<br>
> directio 2M).<br>
> It seems like readv() returns 0, but finalize_request returns NGX_AGAIN.<br>
<br>
</div>ENOPARSE<br>
<br>
The ngx_http_finalize_request() is void, it doesn't return anything.<br>
<br>
You may want to provide debug log to make it clear what are you<br>
talking about.<br></blockquote><div><br></div><div>I'm not exactly sure what part of it is relevant, so instead of posting pages of log I provided steps to reproduce a configuration where nginx shows an erratic behavior, by not outputing the entire response while doing a proxy_pass.</div>
<div><br></div><div>I'll be experimenting more, and I'll let you know if I find the origin of this issue.</div><div><br></div><div>M.</div><div>š</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<font color="#888888"><br>
Maxim Dounin<br>
</font><div><div></div><div class="h5"><br>
> I'm not sure that test is relevant, and what I'm really benching here. And<br>
> if it might not be the tcp layer directly?<br>
> But maybe the failure should be explicit.<br>
><br>
> The behavior was the same with or without the patch. I've also tried various<br>
> combination of proxy buffers, sendfile, directio.<br>
><br>
> Any thoughts ?<br>
><br>
> Thank you!<br>
> Matthieu.<br>
><br>
> On Fri, Oct 14, 2011 at 12:43 PM, Maxim Dounin <<a href="mailto:mdounin@mdounin.ru">mdounin@mdounin.ru</a>> wrote:<br>
><br>
> > Hello!<br>
> ><br>
> > On Wed, Oct 12, 2011 at 09:22:41PM +0400, Maxim Dounin wrote:<br>
> ><br>
> > > Hello!<br>
> > ><br>
> > > On Wed, Oct 12, 2011 at 09:32:41AM -0700, Matthieu Tourne wrote:<br>
> > ><br>
> > > > On Wed, Oct 12, 2011 at 5:20 AM, Maxim Dounin <<a href="mailto:mdounin@mdounin.ru">mdounin@mdounin.ru</a>><br>
> > wrote:<br>
> > > ><br>
> > > > > Hello!<br>
> > > > ><br>
> > > > > On Tue, Oct 11, 2011 at 07:53:36PM -0700, Matthieu Tourne wrote:<br>
> > > > ><br>
> > > > > > Hi all,<br>
> > > > > ><br>
> > > > > > After turning on keepalives, we've been seeing one crash pretty<br>
> > > > > > consistently.<br>
> > > > > > We're running nginx 1.1.5 and here is the backtrace :<br>
> ><br>
> > [...]<br>
> ><br>
> > > Ok, it looks like I'm right and all buffers are in busy chain.<br>
> > > Likely this happens due to upstream sending response in many small<br>
> > > chunks.<br>
> > ><br>
> > > I'll try to reproduce it here and provide proper fix.<br>
> ><br>
> > Please try the attached patch.<br>
> ><br>
> > Maxim Dounin<br>
> ><br>
> > _______________________________________________<br>
> > nginx mailing list<br>
> > <a href="mailto:nginx@nginx.org">nginx@nginx.org</a><br>
> > <a href="http://mailman.nginx.org/mailman/listinfo/nginx" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a><br>
> ><br>
</div></div></blockquote></div><br>