Connection timed out while connecting to upstream, syn packet volume is high

[gtuhl]

I've been troubleshooting this issue on and off the last couple days and
have not been able to eliminate this error from nginx:

2011/12/12 13:12:32 [error] 27810#0: *36209658 connect() failed (110:
Connection timed out) while connecting to upstream

Our setup is a single machine running nginx that serves as reverse proxy
to two separate apache servers.  The setup is handling perhaps a few
dozen requests per second.  The boxes themselves are almost idle, no
issue with load in that respect.

The upstream apache servers are not getting hit hard and are no where
near their maxclient settings.

This error was happening every second or two on the nginx side until I
set "net.ipv4.tcp_syncookies = 1" on the apache machines (not sure why
it wasn't turned out by default).  After making that change the
frequency dropped to 1 or 2 connection time outs per minute.  Similarly,
leaving syncookies off but increasing "net.ipv4.tcp_max_syn_backlog"
alleviated the issue as well, though didn't stop it entirely.

If I use tcdump to count syn packets I see 8x the volume to these
nginx-fronted apache machines that I see for other datacenters handling
far more load (hundreds of concurrent connections) that are behind
traditional hardware load balancers.

Is there a reason nginx would be generating so many syn packets?  Or,
anyone have recommended settings for alleviating it as much as possible
at least?

Any help is greatly appreciated.

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,219955,219955#msg-219955