Nginx + Rails (Thin Server) - Double Bad Gateway Problem

[Maxim Dounin]

Hello!

On Wed, Mar 28, 2012 at 09:26:38AM -0400, djlebersilvestre wrote:

> Hi,
> 
> I'm facing a strange problem using Nginx with reverse proxy to a ruby
> Thin Server. 
> When someone calls an unrouteable path in our Rails App, Nginx launches
> a 502 - Bad Gateway correctly. 
> When Nginx receives another resquest (just after the 502), that is valid
> and routeable, it launches directly again a 502 - Bad Gateway without
> passing the request to the Thin Server (we can check this in the nginx
> logs and the app logs). 
> When I call the same valid URL again, the Nginx passes the call to the
> Thin Server and it returns to normal operation.
> 
> It seems to cache something in Nginx. When this happens and only in the
> third call it gets back to normal track. 
> I've researched the Internet for this issue but nothing conclusive... 
> As a workaround, I've changed the configuration disabiling the
> next_upstream. 
> This resolved my problem, but I've lost the next_upstream feature that
> calls the others servers when one of them fail...

As long as an upstream server fails and triggers 
proxy_next_upstream, the upstream server in question is marked 
dead and won't get any requests till fail_timeout passes, see

http://nginx.org/en/docs/http/ngx_http_upstream_module.html#server

It looks like in your case all servers fail to handle request 
properly and just close connection, hence all of them are marked 
down.  On next request nginx sees that all backends are down, 
complains about "no live upstreams" to error log, returns error 
and marks all backends up again for quick recovery.  This is 
expected behaviour.

Correct solution for your problem is to fix upstream servers to 
properly return HTTP response in case of errors instead of just 
closing connection.  Something like 404 (as long as I see it's 
appropriate in your case) will be a good choice.

Maxim Dounin