nginx-sticky & nginx_http_upstream_check modules not working together

[n1xman]

Hi,

We use nginx-sticky-module for session persistence and we planned to use
nginx_http_upstream_check_module for to check the upstream service
health.

Scenario: We expect nginx_http_upstream_check_module to check the
service (return 2xx|3xx) and take it out from the upstream pool if the
"service" is not responsive. And nginx-sticky should respect the
decision taken by nginx_http_upstream and start new session to available
upstream server. However, the upstream server is up with the service
port; only the content not being served.

The problem in this scenario, Nginx continue to serve the failed
upstream server until we shutdown the upstream service port. Note that
http_upstream module status page indicates the respective service is
down upon I glitch the service.

I think nginx-sticky got the previous cookie and found the peer match
which is the failed server and continue to maintain the previous
session. If I shutdown the server/port, then sticky tries available
peers and create a new session with available upstream server.

What we need is nginx-sticky module to respect the decision taken by the
nginx_http_upstream module as the priority is "service" availability not
just the "server and port" availability. We run jboss on upstream
servers and the probability is very high the service is malfunction but
the jboss server is up. These two modules are working fine
individually.

Here is the configuration I have tried. We really need your support on
possible configuration options to get this two modules work together.

upstream cluster {
  sticky;
  server 192.168.0.61:8081;
  server 192.168.0.61:8082;
        check interval=3000 rise=2 fall=5 timeout=1000 type=http;
        check_http_send "GET /keepalive.html HTTP/1.0\r\n\r\n";
        check_http_expect_alive http_2xx http_3xx;
}

server {
    listen       80;
    server_name  localhost;

    access_log /var/log/nginx/access.log main;
    error_log /var/log/nginx/error.log warn;

location /stat {
    check_status;
}

 location /test {
        proxy_pass http://cluster/service1/test.html;
        error_log /var/log/nginx/error.log debug;
        }

Here is my nginx version:

nginx version: nginx/1.2.1-1
built by gcc 4.1.2 20080704 (Red Hat 4.1.2-48)
TLS SNI support disabled
configure arguments: --prefix=/etc/nginx/ --sbin-path=/usr/sbin/nginx
--conf-path=/etc/nginx/nginx.conf
--error-log-path=/var/log/nginx/error.log
--http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid
--lock-path=/var/run/nginx.lock
--http-client-body-temp-path=/var/cache/nginx/client_temp
--http-proxy-temp-path=/var/cache/nginx/proxy_temp
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp
--http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx
--group=nginx --with-http_ssl_module --with-http_realip_module
--with-http_addition_module --with-http_sub_module
--with-http_dav_module --with-http_flv_module --with-http_mp4_module
--with-http_gzip_static_module --with-http_random_index_module
--with-http_secure_link_module --with-http_stub_status_module
--with-mail_ssl_module --with-file-aio --with-debug --with-cc-opt='-O2
-g -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables'
--without-http_uwsgi_module --without-http_scgi_module
--without-mail_pop3_module --without-mail_imap_module
--without-mail_smtp_module
--add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/simpl-ngx_devel_kit-24202b4
--add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/agentzh-echo-nginx-module-080c0a1
--add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/agentzh-set-misc-nginx-module-87d0ab2
--add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/mikewest-nginx-static-etags-25bfaf9
--add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/nginx-sticky-module-1.0
--add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/agentzh-memc-nginx-module-8befc56
--add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/agentzh-srcache-nginx-module-8df221e
--add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/yaoweibin-nginx_upstream_check_module-dfee401

uname -rop
2.6.18-194.el5PAE i686 GNU/Linux

cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.5 (Tikanga)

debug logs:
service down, service port still up - http://pastebin.com/S9wJFkmR
service down, service port down- http://pastebin.com/HSjMMs75

Thanks in advance.

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,227937,227937#msg-227937