upstream - behavior on pool exhaustion

[Ruslan Ermilov]

On Wed, Apr 19, 2017 at 05:26:17PM +0200, B.R. via nginx wrote:
> On Wed, Apr 19, 2017 at 10:51 AM, Ruslan Ermilov <ru at nginx.com> wrote:
> 
> > And what about the next request when all of servers are either "down"
> > or "unavailable"?
> >
> 
> To me, all the unavailable servers have been 'tried', ie their state has
> been checked, as the docs
> <http://nginx.org/en/docs/http/ngx_http_upstream_module.html#upstream> say
> every server of an upstream group will be checked.
> Thus, they should all appear, along with the time it took for that lookup
> to be made (usually 0.000 since no communication occurred and a simple
> internal check has probably been made for the duration of the 'unavailable'
> state).
> 
> ​No special value whatsoever, as none required, and the grammar of the 2
> variables is stable.​
> ​If an explicit message about attempts against completely unavailable
> upstream groups should be logged, it would be so in the error log.​

Given the following config,

    upstream u {
	server 127.0.0.1:8001;
	server 127.0.0.1:8002;
	server 127.0.0.1:8003;
    }

    log_format u "upstream_addr={$upstream_addr} upstream_status={$upstream_status}";

    server {
	access_log logs/u.log u;

	listen 8000;

	location / {
	    proxy_pass http://u;
	}
    }

when none of the upstream servers are available, the first request will
log

upstream_addr={127.0.0.1:8001, 127.0.0.1:8002, 127.0.0.1:8003} upstream_status={502, 502, 502}

And the next request will log:

upstream_addr={u} upstream_status={502}


If you add the 4th "server 127.0.0.1:8004 down" and start afresh, then the
first request will end up with

upstream_addr={127.0.0.1:8001, 127.0.0.1:8002, 127.0.0.1:8003, u} upstream_status={502, 502, 502, 502}

This is because the number of attempts equals the number of servers by
default.


Again, this is not a bug but a feature.

The number of values in $upstream_* always corresponds to the number
of attempts made to select a server.  The last value can be a special
value with the name of the upstream group when an attempt to select
a peer has failed.  This can happen on the 1st attempt, and then the
only value will be the upstream group name.