SSL_read error on multiple simultaneous upstream SSL downloads

Agent Coulson shield1182 at gmail.com
Fri Oct 18 19:59:40 UTC 2013


Yes, I am able to reproduce this talking to the same nginx as an upstream,
here is my new config.  To reproduce, create a file in the root which is
several Mb, i used 20Mb, and issus multiple simultaneous curl's to the
object, i found rate-limiting my curl is the best way to repro.  This
suggests there is some problem when we have to buffer.  I'm skeptical that
this is an openssl issue as I have used multiple different openssl versions
and still run into this.  However for completeness, I've reprod with
openssl sources from openssl.org (openssl-1.0.1e) as you suggested.

Updated conf:

### Begin ngxin.conf ###

worker_processes  1;

error_log  logs/error.log debug;

pid        logs/nginx.pid;

events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    access_log  logs/access.log;

    keepalive_timeout  60;

    upstream http {
        server 127.0.0.1:1183;
        keepalive 512;
    }

    server {
        listen       1182 default_server;

        server_name  -;

        ssl_protocols SSLv3 TLSv1;
        ssl_ciphers RC4:HIGH:!aNULL:!MD5;
        ssl_prefer_server_ciphers on;

        location / {
            proxy_pass https://http;

            proxy_redirect      off;
            proxy_read_timeout  10s;
            proxy_connect_timeout 6s;

            proxy_buffering     off;
            proxy_buffer_size   64k;
            proxy_buffers       6 16k;
            proxy_busy_buffers_size 80k;

            proxy_pass_header   Server;
            proxy_pass_header   Date;
            proxy_pass_header   X-Pad;

            proxy_set_header    Connection "Keep-Alive";
            proxy_set_header    Host "upstream.srv";
        }
    }

    server {
        listen     1183 ssl;
        server_name upstream.srv;

        ssl_certificate /var/nginx/conf/upstream.srv.pem;
        ssl_certificate_key /var/nginx/conf/upstream.srv.key;

        ssl_protocols SSLv3 TLSv1;
        ssl_ciphers RC4:HIGH:!aNULL:!MD5;
        ssl_prefer_server_ciphers on;

        location / {
            root /var/nginx/html;
        }
    }
}

### End ngxin.conf ###

configure flags for nginx 1.4.3:

./configure --prefix=/var/nginx --with-debug --with-http_ssl_module
--without-http_auth_basic_module --without-http_autoindex_module
--without-http_browser_module --without-http-cache
--without-http_charset_module --without-http_empty_gif_module
--without-http_fastcgi_module --without-http_geo_module
--without-http_gzip_module --without-http_limit_conn_module
--without-http_map_module --without-http_memcached_module
--without-http_referer_module --without-http_rewrite_module
--without-http_scgi_module --without-http_split_clients_module
--without-http_ssi_module --without-http_upstream_ip_hash_module
--without-http_userid_module --without-http_uwsgi_module
--without-mail_imap_module --without-mail_pop3_module
--without-mail_smtp_module --with-openssl=/tmp/openssl-1.0.1e

I start nginx and then issue 3 simultaneous curl's from the local box,
rate-limited.  This should be sufficient for anyone else to repro the issue.

curl --limit-rate 800k -v -o /dev/null http://localhost:1182/20m.txt&
curl --limit-rate 800k -v -o /dev/null http://localhost:1182/20m.txt&
curl --limit-rate 800k -v -o /dev/null http://localhost:1182/20m.txt&

At least one will fail with bytes remaining, and you will see the error in
the error.log.

2013/10/18 19:56:50 [error] 14667#0: *4 SSL_read() failed (SSL:
error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record
mac) while sending to client, client: 127.0.0.1, server: -, request: "GET
/20m.bin HTTP/1.1", upstream: "https://127.0.0.1:1183/20m.bin", host:
"localhost:1182"

thanks for your attention.



On Fri, Oct 18, 2013 at 7:06 PM, Maxim Dounin <mdounin at mdounin.ru> wrote:

> Hello!
>
> On Fri, Oct 18, 2013 at 06:01:14PM +0000, Agent Coulson wrote:
>
> > I am able to reproduce the following error when I have nginx configured
> > with an upstream https connection.  I have tweaked various settings all
> to
> > no avail (proxy_buffer_size, proxy_buffers, proxy_ssl_session_reuse).
> >
> > 2013/10/18 17:17:31 [debug] 15644#0: *39 SSL_read: -1, SSL_pending: 16384
> > 2013/10/18 17:17:31 [debug] 15644#0: *39 SSL_get_error: 1
> > 2013/10/18 17:17:31 [error] 15644#0: *39 SSL_read() failed (SSL:
> > error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad
> record
> > mac) while sending to client, client: 127.0.0.1, server: -, request: "GET
> > /test-1 HTTP/1.1", upstream: "https://x.x.x.x:443/test-1", host:
> > "localhost:1182"
>
> I tend to think it's highly unlikely it's a problem in nginx.
> Most likely, it's a problem either in OpenSSL library used on
> nginx side, or in SSL implementation used on a backend.
>
> First thing I would recommend to test is to make sure you are able
> to reporoduce the problem:
>
> 1. Using nginx statically compiled with a known version of the
> OpenSSL library (--with-openssl=..., with sources from
> openssl.org).
>
> 2. Using the same nginx as a backend.
>
> [...]
>
> > I've seen a bug report on this too (
> http://trac.nginx.org/nginx/ticket/215),
> > so thought i would send this here to see if anyone else is actively
> working
> > on the issue.
>
> As of now, no one provided enough steps to reproduce the problem.
> And, see above, most likely the problem is not in nginx.
>
> --
> Maxim Dounin
> http://nginx.org/en/donation.html
>
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20131018/58a6a3e5/attachment-0001.html>


More information about the nginx-devel mailing list