Websocket tunnel broken with existing SSL session

[Eiríkur Nilsson]

We've been debugging this issue for 3 days now and even though we have a
temporary fix, we're still puzzled about it.

There is an iOS app, which opens a websocket connection to our server over
SSL. Our server runs SmartOS and has nginx 1.5.0 (also happens on 1.4.1)
proxying to a backend server running in NodeJS.

To reproduce, I start my app, a websocket connection is established and
works well, then I put the app to sleep for awhile until nginx kills the
connection. When I reopen the app, the following happens:

1) App notices that the connection is dead and reconnects.
2) Behind the scenes, iOS reuses the SSL session from before and quickly
opens a new socket.
3) A HTTP upgrade request and response flow across with no problems.
4) With a successful web-socket established on both sides, the client
starts sending frames. However, none of these gets delivered to the backend
server.
5) After a minute, nginx kills the connection even though the client is
sending periodic pings.
6) Back to 1.

I haven't managed to reduce the test case or reproduce it in another
environment yet. This only happens when using SSL. In wireshark I see the
websocket frames being sent from the iPhone client and TCP acked properly.

What currently fixes the problem is to disable SSL session reuse in nginx.
Then every websocket connection works like it should.

Here is the config before the fix:
###
    server {
        ### Server port and name ###
        listen          80 default_server;
        listen          443 default_server ssl;
        server_name     test.mydomain.com;

        ### SSL cert files ###
        ssl_certificate         /opt/local/etc/nginx/ssl/certificate.crt;
        ssl_certificate_key     /opt/local/etc/nginx/ssl/certificate.key;

        ### SSL specific settings ###
        ssl_session_cache shared:SSL:10m;
        ssl_session_timeout 10m;
        ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers RC4:HIGH:!aNULL:!MD5;
        ssl_prefer_server_ciphers on;

        keepalive_timeout 60;
        client_max_body_size 10m;

        location / {
            access_log off;
            proxy_pass http://localhost:3003;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

            # WebSocket support (nginx 1.4)
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
        }
    }


Best regards,
Eirikur Nilsson
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20140114/900e5b50/attachment.html>