Achieving strong client -> upstrem_server affinity

[Srirang Doddihal]

Hello Maxim,

Thank you very much for the detailed explanation. Things are much clearer now.

On Fri, Aug 31, 2012 at 2:25 PM, Maxim Dounin <mdounin at mdounin.ru> wrote:
> Hello!
>
> On Thu, Aug 30, 2012 at 11:58:02PM +0530, Srirang Doddihal wrote:
>
>> Hi,
>>
>> I am using Nginx 1.1.19 on Ubuntu 12.04 (LTS) server.
>>
>> Nginx is used to load balance traffic between two instances of the
>> Punjab XMPP-BOSH server.
>>
>> Below is the relevant part from my nginx configuration :
>>
>> upstream chat_bosh {
>>         ip_hash;
>>         server 10.98.29.135:5280;
>>         server 10.98.29.135:5281;
>> }
>>
>> server {
>>  ......................
>> .......................
>>         location /http-bind {
>>                 proxy_next_upstream off;
>>                 proxy_pass http://chat_bosh;
>>                 expires off;
>>         }
>> }
>>
>> I am using ip_hash to make sure that a client will always be served by
>> the same upstream server. This is essential.
>> I am using "proxy_next_upstream off;" to prevent a request being tried
>> on multiple upstream servers, because such requests will invariably
>> fail. I realize that this will cost me redundancy and fallback in case
>> a particular upstream server goes down, but that isn't useful in this
>> particular case. I plan to handle it separately via monitoring and
>> alerts.
>
> In addition to "proxy_next_upstream off" you should at least
> specify max_fails=0 for upstream servers.  Else a server might
> be considered down and request from a client will be re-hashed to
> another one.

Got it. How about the following scenario :

Request - 1]  "client-1" is forwarded to "server-1".
Request - 2]  "server-1" does not respond properly and hence is
considered down. "client-1" gets an error message
Request - 3]  "client-1" is now hashed to "server-2" and is forwarded
to "server-2"
Request - 4]  Now will "client-1" continue to be forwarded to
"server-2" or will it be come back to "server-1"?

i.e Whether the re-hash is permanent or a temporary?

>
> Note that docs might be a bit misleading here as they say one
> should refer to proxy_next_upstream setting to see what is
> considered to be server failure.  This isn't exactly true: if
> upstream server fails to return correct http answer (i.e. on
> error, timeout, invalid_header in proxy_next_upstream terms) the
> failure is always counted.

Understood till here.

> What can be considered to be failure
> or not is valid http responses, i.e. http_500 and so on.
>

This was confusing. Are you saying that any only HTTP 1xx, 2xx or 3xx
responses from the upstream server will not count towards failure
count and any 4xx or 5xx responses will be considered as failures?

>> Anomalies :
>>
>> 1) Despite ip_hash being specified the request from a particular
>> client IP sometimes (close to 7% of requests) get routed to a second
>> upstream server
>
> This is correct as long as upstream servers fail and you don't use
> max_fails=0.  See above.
>
>> 2) Despute proxy_next_upstream off; some requests (about 5%) are tried
>> over multiple upstream servers.
>
> This is strange, and you may want to provide more info, see
> http://wiki.nginx.org/Debugging.
>

I will try to get a debug log. Currently I am using the Ubuntu
package. I probably will have do a custom build for this.

> I would suggest this is likely some configuration error though
> (requests are handled in another location, without
> proxy_next_upstream set to off?).

All requests to the concerned upstream servers are sent from only one
location and that location has proxy_next_upstream set to off.

I am setting up a test environment to isolate this issue. I will get
back with more details a little later.
Is there anything specific that you want to capture?

>
> Maxim Dounin
>
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx



-- 
Regards,
Srirang G Doddihal
Brahmana.

The LIGHT shows the way.
The WISE see it.
The BRAVE walk it.
The PERSISTENT endure and complete it.

I want to do it all ALONE.