Achieving strong client -> upstrem_server affinity

[Srirang Doddihal]

Hi,

I am using Nginx 1.1.19 on Ubuntu 12.04 (LTS) server.

Nginx is used to load balance traffic between two instances of the
Punjab XMPP-BOSH server.

Below is the relevant part from my nginx configuration :

upstream chat_bosh {
        ip_hash;
        server 10.98.29.135:5280;
        server 10.98.29.135:5281;
}

server {
 ......................
.......................
        location /http-bind {
                proxy_next_upstream off;
                proxy_pass http://chat_bosh;
                expires off;
        }
}

I am using ip_hash to make sure that a client will always be served by
the same upstream server. This is essential.
I am using "proxy_next_upstream off;" to prevent a request being tried
on multiple upstream servers, because such requests will invariably
fail. I realize that this will cost me redundancy and fallback in case
a particular upstream server goes down, but that isn't useful in this
particular case. I plan to handle it separately via monitoring and
alerts.

Anomalies :

1) Despite ip_hash being specified the request from a particular
client IP sometimes (close to 7% of requests) get routed to a second
upstream server
2) Despute proxy_next_upstream off; some requests (about 5%) are tried
over multiple upstream servers.

What could be causing these and how do I go about fixing these?

Here are two sets of log line captures which depict the above
mentioned problems.

http://pastebin.com/vnEHQBxK - upstrem_next_server on; (i.e. default value)
http://pastebin.com/vvPBsPgT - - upstrem_next_server off; (as specified above)

These log lines were created with this command : tail -F
/var/log/nginx/access.log | grep "POST /http-bind" | awk '{print $1
"|"  $3 "|" $8 "|" $14}'

$1 = $remote_addr
$2 = $upstream_addr
$3 = $msec
$4 = $status

To see the ip_hash anamoly search for "|404" and look at the adjacent
lines. The same $remote_addr will be forwarded to two different
upstream servers.
To see the upstream_next_server off; anamoly search for "HTTP" -
Because of my brittle awk statement the status is replace by the
string "HTTP/1.1" when upstream_addr has multiple addresses.

--
Regards,
Srirang G Doddihal
Brahmana.

The LIGHT shows the way.
The WISE see it.
The BRAVE walk it.
The PERSISTENT endure and complete it.

I want to do it all ALONE.