Crash (double free or corruption) on trying to proxy to localhost

[Stephen Weeks]

I've currently got a pool of systems running nginx proxying to a pool of
systems running apache.  I'm trying to move to running nginx locally on the
apache hosts instead of separate systems to avoid the extra network hop,
make more-efficient use of resources, and enable some future development
(including migrating to running our application on nginx via fastcgi instead
of apache, ideally).  we've currently got some significant architecture
built up around apache, so converting right now is uncomfortable.

Ideally, I'd like nginx to just serve from localhost, but fail over to the
rest of the pool when localhost in unavailable, so in my upstream I have
every server except for localhost set as 'backup'.  I'm otherwise running
identical configurations of apache and nginx on a single system together as
used in the rest of the two pools.  This works exactly as expected, except
that I get a few crashes of nginx workers every minute.  This only happens
when proxying to the local system.  If I proxy anywhere else, it works
fine.  Other proxies can serve from this system without trouble.  I see this
same behaviour on other hosts when I build them the same way, so it's not an
error with the host.  I see this crash on 0.7.65, 0.8.54, and 1.0.0, running
on Ubuntu 10.04 LTS.  I see this crash whether I'm connecting to 127.0.0.1
or the host's local IP.  I see this crash whether I'm listening on *:80 or
<public ip>:80.  I see this crash whether I'm connecting to :80 or running
apache on a different port and connecting to :81.  I see this crash whether
I'm running ubuntu's "nginx-light" configuration, or their "nginx-full"
configuration.  I see no errors logged from apache.

1) I'd really love to make this work, so if there's anything else I can try,
any additional debugging information I can give, I'd appreciate it.
2) Nginx has been very useful to me so far, so I thought you'd appreciate a
bug report.

Posted on github, I have a problem description, section of a debug log, my
(slightly edited: flattened includes and stripped an IP) nginx.conf, a gdb
backtrace, and some additional information I was asked for when looking for
help on IRC.  This is everything I've been able to come up with that sounds
plausibly relevant.

https://gist.github.com/1574dbaf3a3dcda920a2

Any help?

(Thanks to MTechnology and kolbyjack for helping me troubleshoot this on
IRC)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://nginx.org/pipermail/nginx/attachments/20110501/9d18eb10/attachment.html>