proxy_upstream_next while no live upstreams

[Maxim Dounin]

Hello!

On Fri, May 12, 2017 at 01:24:14PM +0800, Wu Bingzheng wrote:

> 
> The last request before this 502 request is almost 20 minutes ago and its response code is 200.
> 
> The proxy_next_upstream conf:
>     proxy_next_upstream error timeout invalid_header http_502 http_503 http_504;
> 
> Here is the access log. The upstream server 192.168.0.6 is DOWN. The line-10 is the 502 request:
> 
>   1 [03/May/2017:14:35:38 -0400] "POST /x/y HTTP/1.1" 200  "192.168.0.6:8181, 192.168.0.5:8181" 0.012 0.001, 0.011
>   2 [03/May/2017:14:35:38 -0400] "POST /x/y HTTP/1.1" 200  "192.168.0.5:8181" 0.013 0.013
>   3 [03/May/2017:14:54:30 -0400] "POST /x/y HTTP/1.1" 200  "192.168.0.5:8181" 0.206 0.206
>   4 [03/May/2017:15:03:08 -0400] "POST /x/y HTTP/1.1" 200  "192.168.0.5:8181" 0.154 0.154
>   5 [03/May/2017:15:40:51 -0400] "POST /x/y HTTP/1.1" 200  "192.168.0.6:8181, 192.168.0.5:8181" 0.012 0.000, 0.012
>   6 [03/May/2017:15:40:51 -0400] "POST /x/y HTTP/1.1" 200  "192.168.0.5:8181" 0.014 0.014
>   7 [03/May/2017:15:40:51 -0400] "POST /x/y HTTP/1.1" 200  "192.168.0.5:8181" 0.016 0.016
>   8 [03/May/2017:15:40:51 -0400] "POST /x/y HTTP/1.1" 200  "192.168.0.5:8181" 0.017 0.017
>   9 [03/May/2017:15:40:51 -0400] "POST /x/y HTTP/1.1" 200  "192.168.0.5:8181" 0.011 0.011
>  10 [03/May/2017:15:59:06 -0400] "POST /x/y HTTP/1.1" 502  "192.168.0.6:8181, test_backend" 0.000 0.000, 0.000
>  11 [03/May/2017:15:59:07 -0400] "POST /x/y HTTP/1.1" 200  "10.255.222.206:8181" 0.260 0.260

Looking into response status code in access logs is not enough to 
understand if a server is up or down.  For at least the following 
reasons:

- there might be over requests currently in flight which are not 
  yet logged;

- errors may occur while sending response body, and hence status 
  code will not show if there was an error.

It is usually a good idea to look into error logs instead.

-- 
Maxim Dounin
http://nginx.org/