Error handling of "Connection refused" conditions

[Dustin Wenz]

I am using nginx to distribute http load across three upstream application servers. It actually works really well in normal usage, but when we restart one of the upstream servers for maintenance (such as for an OS update), nginx tends to hang incoming requests for some time (30 seconds or more). When a node is brought offline, the nginx error logs show a few messages like this:

	upstream prematurely closed connection while reading response header from upstream

Followed by many messages like this (these last basically as long as the node is offline):

	kevent() reported that connect() failed (61: Connection refused) while connecting to upstream

This seems ok to me, since the upstream server did go away. What should I expect to see of incoming http requests when this happens? Shouldn't nginx route requests to the remaining application servers, making the outage invisible to users? If it's expected to hang some requests when this happens, what timeout could be adjusted to minimize that duration?

Thanks,

	- .Dustin