[nginx] SSL: avoid calling SSL_shutdown() during handshake (ticket #901).

[Maxim Dounin]

details:   http://hg.nginx.org/nginx/rev/062c189fee20
branches:  
changeset: 6407:062c189fee20
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Fri Feb 19 17:27:30 2016 +0300
description:
SSL: avoid calling SSL_shutdown() during handshake (ticket #901).

This fixes "called a function you should not call" and
"shutdown while in init" errors as observed with OpenSSL 1.0.2f
due to changes in how OpenSSL handles SSL_shutdown() during
SSL handshakes.

diffstat:

 src/event/ngx_event_openssl.c |  13 +++++++++++++
 1 files changed, 13 insertions(+), 0 deletions(-)

diffs (23 lines):

diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c
--- a/src/event/ngx_event_openssl.c
+++ b/src/event/ngx_event_openssl.c
@@ -1767,6 +1767,19 @@ ngx_ssl_shutdown(ngx_connection_t *c)
     int        n, sslerr, mode;
     ngx_err_t  err;
 
+    if (SSL_in_init(c->ssl->connection)) {
+        /*
+         * OpenSSL 1.0.2f complains if SSL_shutdown() is called during
+         * an SSL handshake, while previous versions always return 0.
+         * Avoid calling SSL_shutdown() if handshake wasn't completed.
+         */
+
+        SSL_free(c->ssl->connection);
+        c->ssl = NULL;
+
+        return NGX_OK;
+    }
+
     if (c->timedout) {
         mode = SSL_RECEIVED_SHUTDOWN|SSL_SENT_SHUTDOWN;
         SSL_set_quiet_shutdown(c->ssl->connection, 1);