Achieving strong client -> upstrem_server affinity

[Maxim Dounin]

Hello!

On Thu, Aug 30, 2012 at 11:58:02PM +0530, Srirang Doddihal wrote:

> Hi,
> 
> I am using Nginx 1.1.19 on Ubuntu 12.04 (LTS) server.
> 
> Nginx is used to load balance traffic between two instances of the
> Punjab XMPP-BOSH server.
> 
> Below is the relevant part from my nginx configuration :
> 
> upstream chat_bosh {
>         ip_hash;
>         server 10.98.29.135:5280;
>         server 10.98.29.135:5281;
> }
> 
> server {
>  ......................
> .......................
>         location /http-bind {
>                 proxy_next_upstream off;
>                 proxy_pass http://chat_bosh;
>                 expires off;
>         }
> }
> 
> I am using ip_hash to make sure that a client will always be served by
> the same upstream server. This is essential.
> I am using "proxy_next_upstream off;" to prevent a request being tried
> on multiple upstream servers, because such requests will invariably
> fail. I realize that this will cost me redundancy and fallback in case
> a particular upstream server goes down, but that isn't useful in this
> particular case. I plan to handle it separately via monitoring and
> alerts.

In addition to "proxy_next_upstream off" you should at least 
specify max_fails=0 for upstream servers.  Else a server might 
be considered down and request from a client will be re-hashed to 
another one.

Note that docs might be a bit misleading here as they say one 
should refer to proxy_next_upstream setting to see what is 
considered to be server failure.  This isn't exactly true: if 
upstream server fails to return correct http answer (i.e. on 
error, timeout, invalid_header in proxy_next_upstream terms) the 
failure is always counted.  What can be considered to be failure 
or not is valid http responses, i.e. http_500 and so on.

> Anomalies :
> 
> 1) Despite ip_hash being specified the request from a particular
> client IP sometimes (close to 7% of requests) get routed to a second
> upstream server

This is correct as long as upstream servers fail and you don't use 
max_fails=0.  See above.

> 2) Despute proxy_next_upstream off; some requests (about 5%) are tried
> over multiple upstream servers.

This is strange, and you may want to provide more info, see 
http://wiki.nginx.org/Debugging.

I would suggest this is likely some configuration error though 
(requests are handled in another location, without 
proxy_next_upstream set to off?).

Maxim Dounin