[nginx] QUIC: better approach for premature handshake completion.

[noreply at nginx.com]

details:   https://github.com/nginx/nginx/commit/1d4d2f2c962c33aafdd8f79d9fc50b7cacf05e24
branches:  master
commit:    1d4d2f2c962c33aafdd8f79d9fc50b7cacf05e24
user:      Sergey Kandaurov <pluknet at nginx.com>
date:      Fri, 16 May 2025 01:10:11 +0400
description:
QUIC: better approach for premature handshake completion.

Using SSL_in_init() to inspect a handshake state was replaced with
SSL_is_init_finished().  This represents a more complete fix to the
BoringSSL issue addressed in 22671b37e.

This provides awareness of the early data handshake state when using
OpenSSL 3.5 TLS callbacks in 0-RTT enabled configurations, which, in
particular, is used to avoid premature completion of the initial TLS
handshake, before required client handshake messages are received.

This is a non-functional change when using BoringSSL.  It supersedes
testing non-positive SSL_do_handshake() results in all supported SSL
libraries, hence simplified.

In preparation for using OpenSSL 3.5 TLS callbacks.

---
 src/event/quic/ngx_event_quic_ssl.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/event/quic/ngx_event_quic_ssl.c b/src/event/quic/ngx_event_quic_ssl.c
index fc8ebd8cf..6ce926c81 100644
--- a/src/event/quic/ngx_event_quic_ssl.c
+++ b/src/event/quic/ngx_event_quic_ssl.c
@@ -463,7 +463,7 @@ ngx_quic_handshake(ngx_connection_t *c)
         }
     }
 
-    if (n <= 0 || SSL_in_init(ssl_conn)) {
+    if (!SSL_is_init_finished(ssl_conn)) {
         if (ngx_quic_keys_available(qc->keys, NGX_QUIC_ENCRYPTION_EARLY_DATA, 0)
             && qc->client_tp_done)
         {