how to make nginx loadbalancer give 404 when all upstream servers are down

[karkunpavan]

Hi Fracis,

Thanks for taking a look. With it curl behaves as expected, but with tools
like ab and httperf, I am not seeing bad gateway responses when both the
upstream servers are down. Read ahead for more details.

Below is info using ab request and curl request.

ab:

When one upstream server is up:

nginx console:
<<client_IP>> - - [14/Aug/2015:13:44:57 +0000] "GET / HTTP/1.0" 200 45 "-"
"ApacheBench/2.3"
<<client_IP>> - - [14/Aug/2015:13:44:57 +0000] "GET / HTTP/1.0" 200 45 "-"
"ApacheBench/2.3"
upstream httpd server console:
<<nginx_IP>> - - [14/Aug/2015:13:44:57 +0000] "GET / HTTP/1.0" 200 45
<<nginx_IP>> - - [14/Aug/2015:13:44:57 +0000] "GET / HTTP/1.0" 200 45


When both upstream servers are down:

2015/08/14 13:43:45 [error] 5#5: *14 connect() failed (111: Connection
refused) while connecting to upstream, client: <<client_IP>>, server: ,
request: "GET / HTTP/1.0", upstream: "http://<upstream_1_IP>>:<port>/",
host: "<nginx_IP:<port>"
2015/08/14 13:43:45 [error] 5#5: *14 connect() failed (111: Connection
refused) while connecting to upstream, client: <<client_IP>>, server: ,
request: "GET / HTTP/1.0", upstream: "http://<upstream_2_IP>:<port>/", host:
"<nginx_IP:<port>"
2015/08/14 13:43:45 [error] 5#5: *14 no live upstreams while connecting to
upstream, client: <<client_IP>>, server: , request: "GET / HTTP/1.0",
upstream: "http://ab_backend/", host: "<nginx_IP:<port>"
<<client_IP>> - - [14/Aug/2015:13:43:45 +0000] "GET / HTTP/1.0" 502 172 "-"
"ApacheBench/2.3"
<<client_IP>> - - [14/Aug/2015:13:43:45 +0000] "GET / HTTP/1.0" 502 172 "-"
"ApacheBench/2.3"

as a result in bith cases ab just shows all successful responses:
# ab -n 1 -c 1 http://<nginx_IP>:<port>/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking <<nginx_IP>> (be patient).....done
Server Software:        nginx/1.9.3
Server Hostname:        10.107.53.127
Server Port:            9000
Document Path:          /
Document Length:        45 bytes
Concurrency Level:      1
Time taken for tests:   0.040 seconds
Complete requests:      1
Failed requests:        0
Write errors:           0
Keep-Alive requests:    1
Total transferred:      285 bytes
HTML transferred:       45 bytes
Requests per second:    24.85 [#/sec] (mean)
Time per request:       40.234 [ms] (mean)
Time per request:       40.234 [ms] (mean, across all concurrent requests)
Transfer rate:          6.92 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        2    2   0.0      2       2
Processing:    39   39   0.0     39      39
Waiting:       38   38   0.0     38      38
Total:         40   40   0.0     40      40


curl:

when one upstream is up:
<<client_IP>> - - [14/Aug/2015:13:58:37 +0000] "GET / HTTP/1.1" 200 45 "-"
"curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0
zlib/1.2.3 libidn/1.18 libssh2/1.2.2"

When both upstream servers are down:
2015/08/14 14:00:37 [error] 8#8: *26 connect() failed (111: Connection
refused) while connecting to upstream, client: <<client_IP>>, server: ,
request: "GET / HTTP/1.1", upstream: "http://<upstream_1_IP>:<port>/", host:
"<nginx_IP>:<port>"
2015/08/14 14:00:37 [error] 8#8: *26 connect() failed (111: Connection
refused) while connecting to upstream, client: <<client_IP>>, server: ,
request: "GET / HTTP/1.1", upstream: "http://<upstream_2_IP>:<port>/", host:
"<nginx_IP>:<port>"
<<client_IP>> - - [14/Aug/2015:14:00:37 +0000] "GET / HTTP/1.1" 502 172 "-"
"curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0
zlib/1.2.3 libidn/1.18 libssh2/1.2.2"

and curl gets a "502 Bad Gateway" which is right.

Any suggestions what I might be doing wrong?

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,260980,260998#msg-260998