502 Bad Gateway/while SSL handshaking to upstream

[teege]

I have been fighting with this one for some time, would appreciate
some assistance.

in my test environment on ec2, I have:

[ nginx -> haproxy ] -> [ apache w/ ajp -> tomcat ] -> [ mysql cluster ]

nginx and haproxy on the same machine, apache and tomcat on the same
machine - and the mysql cluster has 2-4 sql nodes+data nodes.

I have been testing with a single GET request, which exercises all of
the above (ex. ab -n 200 -c 200). the net of the problem is that 95%
of the time, 1/3 to 1/2 of my connections result in a 502 returned by
nginx. I've yet to have the full 200 completed requests.

the nginx error.log contains both types of errors below, but there is
a direct relation to the failures - if 93/200 connections result in a
502, I will have exactly 93 SSL handshake errors - so I've narrowed
the problem down I believe.

a)
2010/04/23 07:49:43 [error] 18430#0: *364 SSL_do_handshake() failed
(SSL: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or
bad record mac) while SSL handshaking to upstream, client: 174.1.X.X,
server: api.example.com, request: "GET /1.0/getName HTTP/1.0",
upstream: "https://127.0.0.1:8003/1.0/getName", host:
"api.example.com"

b)
2010/04/23 07:49:41 [info] 18430#0: *260 peer closed connection in SSL
handshake while SSL handshaking to upstream, client: 174.10.X.X,
server: api.example.com, request: "GET /1.0/getName HTTP/1.0",
upstream: "https://127.0.0.1:8003/1.0/getName", host:
"api.example.com"

unfortunately, it's unclear why this is happening. nginx config:

  server {
      listen 0.0.0.0:443;
      server_name api.example.com;

      location ~ /* {
        proxy_pass         https://api_443;
      }

      access_log            logs/api.mobioid.com-access.log main;

      ssl                   on;
      keepalive_timeout     70;
      ssl_certificate       /u/nginx/ssl/api.example.com.crt;
      ssl_certificate_key   /u/nginx/ssl/api.example.com.key;
  }

    upstream api_443   { server 127.0.0.1:8003; }

and haproxy config on port 8003:

  listen api_443 127.0.0.1:8003
    mode    tcp
    option  ssl-hello-chk
        server s1.example.com 10.242.X.X:443 weight 1 minconn 3
maxconn 125 check inter 20000

api.example.com is a trusted cert, s1.example.com/apache is a self-signed cert.

a previous email thread by David Taveras last month, asked about
essentially the same problem:

	http://nginx.org/pipermail/nginx/2010-March/019120.html

and my config is also similar. unfortunately no one had responded to his query.

I'm hopeful the community will be able to lend a hand here!

teege