[nginx] SSL: workaround for incorrect SSL_write() errors in OpenSSL 1.1.1.

[Maxim Dounin]

details:   https://hg.nginx.org/nginx/rev/61011bfcdb49
branches:  
changeset: 7706:61011bfcdb49
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Wed Sep 16 18:26:22 2020 +0300
description:
SSL: workaround for incorrect SSL_write() errors in OpenSSL 1.1.1.

OpenSSL 1.1.1 fails to return SSL_ERROR_SYSCALL if an error happens
during SSL_write() after close_notify alert from the peer, and returns
SSL_ERROR_ZERO_RETURN instead.  Broken by this commit, which removes
the "i == 0" check around the SSL_RECEIVED_SHUTDOWN one:

https://git.openssl.org/?p=openssl.git;a=commitdiff;h=8051ab2

In particular, if a client closed the connection without reading
the response but with properly sent close_notify alert, this resulted in
unexpected "SSL_write() failed while ..." critical log message instead
of correct "SSL_write() failed (32: Broken pipe)" at the info level.

Since SSL_ERROR_ZERO_RETURN cannot be legitimately returned after
SSL_write(), the fix is to convert all SSL_ERROR_ZERO_RETURN errors
after SSL_write() to SSL_ERROR_SYSCALL.

diffstat:

 src/event/ngx_event_openssl.c |  12 ++++++++++++
 1 files changed, 12 insertions(+), 0 deletions(-)

diffs (22 lines):

diff -r 3781de64e747 -r 61011bfcdb49 src/event/ngx_event_openssl.c
--- a/src/event/ngx_event_openssl.c	Wed Sep 09 19:26:27 2020 +0300
+++ b/src/event/ngx_event_openssl.c	Wed Sep 16 18:26:22 2020 +0300
@@ -2573,6 +2573,18 @@ ngx_ssl_write(ngx_connection_t *c, u_cha
 
     sslerr = SSL_get_error(c->ssl->connection, n);
 
+    if (sslerr == SSL_ERROR_ZERO_RETURN) {
+
+        /*
+         * OpenSSL 1.1.1 fails to return SSL_ERROR_SYSCALL if an error
+         * happens during SSL_write() after close_notify alert from the
+         * peer, and returns SSL_ERROR_ZERO_RETURN instead,
+         * https://git.openssl.org/?p=openssl.git;a=commitdiff;h=8051ab2
+         */
+
+        sslerr = SSL_ERROR_SYSCALL;
+    }
+
     err = (sslerr == SSL_ERROR_SYSCALL) ? ngx_errno : 0;
 
     ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0, "SSL_get_error: %d", sslerr);