nginx trying to connect to upstream host which is down

[Ravitezu]

Hi, 

Note: I had to change the hostnames and domain names.

nginx version: nginx/1.4.6 (Ubuntu)

I have the following the configuration: 
    upstream ssl-app-cluster {
        ip_hash;
        server app01.example.com:8443 max_fails=1 fail_timeout=60s;
        server app02.example.com:8443 max_fails=1 fail_timeout=60s;
    }

And during the Application deployment(rolling deployment) to those backend
servers(app0{1,2}.example.com) the port 8443 will not be available for
60sec. So, during the deployment I tried hitting the server(api.example.com)
continuously to know, how nginx is routing the traffic, when one of the
servers in down. As I am using ip_hash, my ip is bound to app01.example.com
initially.

Deployment process: 
----------------------------
1. I am running the curl command to hit the server api.example.com in a for
loop and I am being served by app01(I see this on error log with debug
enabled).
2. Deployment process has taken down the app02 host for upgrading the
application on it. This doesn't effect anything, as my IP is bound to app01 
and I am being served by app01. 
3. Now, the  deployment has taken down app01 for deployment. So, I see
there's a "111 connection refused error" and nginx tried to connect to "http
next upstream" which is app02. which is successful. 
4. But, nginx has to wait for 60sec(fail_timeout) to establish connection to
app01, But I see nginx is trying to connect to app01 immediately and I see
there's an error: "111 connection refused error" again and then it
connecting to app02.

Can someone please, tell me why this is happening and how can I change this?


Thanks,
Ravi

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,257139,257139#msg-257139