Upstream Consistent Hash Inconsistency

[Maxim Dounin]

Hello!

On Wed, Feb 10, 2021 at 04:21:20PM +0200, Ari Croock wrote:

> Hi all,
> 
> Not sure if this belongs on the devel mailing list, I thought I would try
> to post here first.
> 
> 
> *Summary*
> The "hash" and "hash ... consistent" upstream directives behave differently
> when an upstream hostname resolves to multiple IP addresses.
> 
> 
> *My Use-case:*
> 
> I am using nginx as a reverse proxy for load balancing inside a
> docker-compose network.
> In docker-compose, you can setup a service with multiple replicas and with
> a common hostname. A DNS lookup returns the IP addresses for each replica
> (e.g. if I have containers "dash_1" and "dash_2", "dash" could resolve to
> the IP addresses for both).
> 
> The application we are hosting requires sticky sessions. We have
> implemented this using the "hash ... consistent" upstream directive.
> So an example upstream could look like:
> upstream test_upstream {
> hash $token consistent;
> server dashboard_test:3838;
> }
> where "$token" is the unique token for hashing.
> Note that "dashboard_test" can resolve to multiple IP addresses as
> explained above.
> 
> As I understand it, when nginx starts up it will do a DNS lookup for
> "dashboard_test" which resolves to multiple IP addresses, effectively
> resulting in an upstream group with multiple "servers".
> 
> 
> *My Problem:*
> 
> It seems that the "hash" and "hash consistent" directives behave
> differently.
> 
> I had a quick look at the source (ngx_http_upstream_hash_module.c) and it
> looks like the "regular" hash method determines an actual IP address from
> the hash, but the "chash" *only determines the entry in the upstream group*.
> 
> Empirically, this does seem to be the case. With the "hash" directive nginx
> always proxies to the same upstream server. With the "hash consistent"
> directive I get a different upstream server on every request.
> 
> Can anyone comment on whether this is intended behaviour? From my point of
> view this seems to be a bug. But I can imagine that changing this behaviour
> might break someone else's use case.

It is intended behaviour.  Consistent hashing depends on the name 
server name written in the configuration to identify servers (and 
distribute requests).  Servers with the same name are effectively 
identical, even if IP address of the server changes.  If the name 
resolves to multiple addresses, these addresses are used in 
round-robin.

That is, your configuration with just one server in the upstream 
block is effectively identical to round-robin configuration.  If 
you want consistent hashing to work, you have to list your backend 
servers in the upstream block by using separate "server" 
directives.

-- 
Maxim Dounin
http://mdounin.ru/