[nginx] QUIC: fixed ssl_reject_handshake error handling.

[noreply at nginx.com]

details:   https://github.com/nginx/nginx/commit/eb5ebbbed74c8ce72465bb079bde0ad29966d170
branches:  master
commit:    eb5ebbbed74c8ce72465bb079bde0ad29966d170
user:      Sergey Kandaurov <pluknet at nginx.com>
date:      Wed, 10 Sep 2025 17:25:36 +0400
description:
QUIC: fixed ssl_reject_handshake error handling.

This was broken in 7468a10b6 (1.29.0), resulting in a missing diagnostics
and SSL error queue not cleared for SSL handshakes rejected by SNI, seen
as "ignoring stale global SSL error" alerts, for instance, when doing SSL
shutdown of a long standing connection after rejecting another one by SNI.

The fix is to move the qc->error check after c->ssl->handshake_rejected is
handled first, to make the error queue cleared.  Although not practicably
visible as needed, this is accompanied by clearing the error queue under
the qc->error case as well, to be on the safe side.

As an implementation note, due to the way of handling invalid transport
parameters for OpenSSL 3.5 and above, which leaves a passed pointer not
advanced on error, SSL_get_error() may return either SSL_ERROR_WANT_READ
or SSL_ERROR_WANT_WRITE depending on a library.  To cope with that, both
qc->error and c->ssl->handshake_rejected checks were moved out of
"sslerr != SSL_ERROR_WANT_READ".

Also, this reconstructs a missing "SSL_do_handshake() failed" diagnostics
for the qc->error case, replacing using ngx_ssl_connection_error() with
ngx_connection_error().  It is made this way to avoid logging at the crit
log level because qc->error set is expected to have an empty error queue.

Reported and tested by Vladimir Homutov.

---
 src/event/quic/ngx_event_quic_ssl.c | 27 ++++++++++++++++-----------
 1 file changed, 16 insertions(+), 11 deletions(-)

diff --git a/src/event/quic/ngx_event_quic_ssl.c b/src/event/quic/ngx_event_quic_ssl.c
index e961c80cd..355348406 100644
--- a/src/event/quic/ngx_event_quic_ssl.c
+++ b/src/event/quic/ngx_event_quic_ssl.c
@@ -695,30 +695,35 @@ ngx_quic_handshake(ngx_connection_t *c)
 
     ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0, "SSL_do_handshake: %d", n);
 
-    if (qc->error) {
-        return NGX_ERROR;
-    }
-
     if (n <= 0) {
         sslerr = SSL_get_error(ssl_conn, n);
 
         ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0, "SSL_get_error: %d",
                        sslerr);
 
-        if (sslerr != SSL_ERROR_WANT_READ) {
-
-            if (c->ssl->handshake_rejected) {
-                ngx_connection_error(c, 0, "handshake rejected");
-                ERR_clear_error();
+        if (c->ssl->handshake_rejected) {
+            ngx_connection_error(c, 0, "handshake rejected");
+            ERR_clear_error();
+            return NGX_ERROR;
+        }
 
-                return NGX_ERROR;
-            }
+        if (qc->error) {
+            ngx_connection_error(c, 0, "SSL_do_handshake() failed");
+            ERR_clear_error();
+            return NGX_ERROR;
+        }
 
+        if (sslerr != SSL_ERROR_WANT_READ) {
             ngx_ssl_connection_error(c, sslerr, 0, "SSL_do_handshake() failed");
             return NGX_ERROR;
         }
     }
 
+    if (qc->error) {
+        ngx_connection_error(c, 0, "SSL_do_handshake() failed");
+        return NGX_ERROR;
+    }
+
     if (!SSL_is_init_finished(ssl_conn)) {
         if (ngx_quic_keys_available(qc->keys, NGX_QUIC_ENCRYPTION_EARLY_DATA, 0)
             && qc->client_tp_done)