I should be able to try this out tonight (>12 hours from now). I'll let you know how it works. Thanks for looking into this for me.<br><br><div class="gmail_quote">On Tue, May 3, 2011 at 4:46 AM, Maxim Dounin <span dir="ltr"><<a href="mailto:mdounin@mdounin.ru">mdounin@mdounin.ru</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Hello!<br>
<div class="im"><br>
On Mon, May 02, 2011 at 04:45:13PM -0700, Stephen Weeks wrote:<br>
<br>
> Sure! I've added it to the post on github. I've slightly edited the log<br>
> out of paranoia (replacing a customer ID and auth key).<br>
><br>
> I can confirm now that I ran this proxy with the upstreamfair module<br>
> overnight and it didn't crash at all. I can't get it to preferentially<br>
> serve to localhost, though, as it doesn't support 'backup' as a server<br>
> attribute, and doesn't seem to really use the weights, so that's suboptimal.<br>
><br>
> Anything else I can add to help troubleshooting this? Anything you'd like<br>
> from the core dump?<br>
<br>
</div>Ok, thank you, it looks like I see the problem.<br>
<br>
Allocation for "tried" flags doesn't take into account number of<br>
backup servers, and if there are more backup servers than normal<br>
ones (and backup servers are in fact used) - this may cause memory<br>
corruption.<br>
<br>
Please try the attached patch.<br>
<font color="#888888"><br>
Maxim Dounin<br>
</font><div><div></div><div class="h5"><br>
><br>
> On Mon, May 2, 2011 at 2:04 PM, Maxim Dounin <<a href="mailto:mdounin@mdounin.ru">mdounin@mdounin.ru</a>> wrote:<br>
><br>
> > Hello!<br>
> ><br>
> > On Sun, May 01, 2011 at 05:54:17PM -0700, Stephen Weeks wrote:<br>
> ><br>
> > > I've currently got a pool of systems running nginx proxying to a pool of<br>
> > > systems running apache. I'm trying to move to running nginx locally on<br>
> > the<br>
> > > apache hosts instead of separate systems to avoid the extra network hop,<br>
> > > make more-efficient use of resources, and enable some future development<br>
> > > (including migrating to running our application on nginx via fastcgi<br>
> > instead<br>
> > > of apache, ideally). we've currently got some significant architecture<br>
> > > built up around apache, so converting right now is uncomfortable.<br>
> > ><br>
> > > Ideally, I'd like nginx to just serve from localhost, but fail over to<br>
> > the<br>
> > > rest of the pool when localhost in unavailable, so in my upstream I have<br>
> > > every server except for localhost set as 'backup'. I'm otherwise running<br>
> > > identical configurations of apache and nginx on a single system together<br>
> > as<br>
> > > used in the rest of the two pools. This works exactly as expected,<br>
> > except<br>
> > > that I get a few crashes of nginx workers every minute. This only<br>
> > happens<br>
> > > when proxying to the local system. If I proxy anywhere else, it works<br>
> > > fine. Other proxies can serve from this system without trouble. I see<br>
> > this<br>
> > > same behaviour on other hosts when I build them the same way, so it's not<br>
> > an<br>
> > > error with the host. I see this crash on 0.7.65, 0.8.54, and 1.0.0,<br>
> > running<br>
> > > on Ubuntu 10.04 LTS. I see this crash whether I'm connecting to<br>
> > 127.0.0.1<br>
> > > or the host's local IP. I see this crash whether I'm listening on *:80<br>
> > or<br>
> > > <public ip>:80. I see this crash whether I'm connecting to :80 or<br>
> > running<br>
> > > apache on a different port and connecting to :81. I see this crash<br>
> > whether<br>
> > > I'm running ubuntu's "nginx-light" configuration, or their "nginx-full"<br>
> > > configuration. I see no errors logged from apache.<br>
> > ><br>
> > > 1) I'd really love to make this work, so if there's anything else I can<br>
> > try,<br>
> > > any additional debugging information I can give, I'd appreciate it.<br>
> > > 2) Nginx has been very useful to me so far, so I thought you'd appreciate<br>
> > a<br>
> > > bug report.<br>
> > ><br>
> > > Posted on github, I have a problem description, section of a debug log,<br>
> > my<br>
> > > (slightly edited: flattened includes and stripped an IP) nginx.conf, a<br>
> > gdb<br>
> > > backtrace, and some additional information I was asked for when looking<br>
> > for<br>
> > > help on IRC. This is everything I've been able to come up with that<br>
> > sounds<br>
> > > plausibly relevant.<br>
> > ><br>
> > > <a href="https://gist.github.com/1574dbaf3a3dcda920a2" target="_blank">https://gist.github.com/1574dbaf3a3dcda920a2</a><br>
> > ><br>
> > > Any help?<br>
> ><br>
> > Could you please provide:<br>
> ><br>
> > 1. nginx -V output<br>
> ><br>
> > 2. Full debug log for '*60' connection (the one which triggered<br>
> > abort in glibc), the one you provided contatins only last part of<br>
> > the connection in question. Running grep -F ' 27772#0: *60 ' on<br>
> > original debug log should produce something useable.<br>
> ><br>
> > Maxim Dounin<br>
> ><br>
> > _______________________________________________<br>
> > nginx mailing list<br>
> > <a href="mailto:nginx@nginx.org">nginx@nginx.org</a><br>
> > <a href="http://nginx.org/mailman/listinfo/nginx" target="_blank">http://nginx.org/mailman/listinfo/nginx</a><br>
> ><br>
<br>
> _______________________________________________<br>
> nginx mailing list<br>
> <a href="mailto:nginx@nginx.org">nginx@nginx.org</a><br>
> <a href="http://nginx.org/mailman/listinfo/nginx" target="_blank">http://nginx.org/mailman/listinfo/nginx</a><br>
<br>
</div></div><br>_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org">nginx@nginx.org</a><br>
<a href="http://nginx.org/mailman/listinfo/nginx" target="_blank">http://nginx.org/mailman/listinfo/nginx</a><br>
<br></blockquote></div><br>