SSL_shutdown() failed (SSL: ... bad write retry)

vergil nginx-forum at forum.nginx.org
Tue Aug 18 09:35:16 UTC 2020


Maxim Dounin Wrote:
-------------------------------------------------------
> Hello!
> 
> On Fri, Aug 14, 2020 at 10:34:49AM -0400, vergil wrote:
> 
> > Maxim Dounin Wrote:
> > -------------------------------------------------------
> > > Hello!
> > > 
> > > On Thu, Aug 13, 2020 at 12:11:54PM -0400, vergil wrote:
> > > 
> > > > Maxim Dounin Wrote:
> > > > -------------------------------------------------------
> > > > > Hello!
> > > > > 
> > > > > On Thu, Aug 13, 2020 at 11:39:36AM -0400, vergil wrote:
> > > > > 
> > > > > > This one was hard to catch.
> > > > > > 
> > > > > > I've captured one error with 30 seconds delta before and
> after
> > > the
> > > > > event.
> > > > > > Where can i attach log file for you? There's 400K messages,
> so i
> > > > > cannot
> > > > > > simple put it here.
> > > > > 
> > > > > Attaching the log to the message into the mailing list should 
> > > > > work, but I'm not sure it's supported by the (obsolete) forum 
> > > > > interface you are using.  If not, you may put the log at a 
> > > > > convinient place and provide a link here, or attach it to a 
> > > > > ticket on trac.nginx.org, or email to me privetely.
> > > > 
> > > > I've attached log file to our S3 public storage. You can
> download it
> > > through
> > > > this link:
> > > >
> > >
> https://drive-public-eu.s3.eu-central-1.amazonaws.com/nginx/nginx-debu
> > > g.csv
> > > > 
> > > > A note: this is a CSV format from our logging system. I can try
> to
> > > extract
> > > > logs in original format if you need.
> > > 
> > > Thanks, but this doesn't seem to contain anything related to the 
> > > SSL_shutdown() except the message itself:
> > > 
> > > "2020-08-13T15:19:03.279Z","7","shmtx lock",
> > > "2020-08-13T15:19:03.279Z","7","shmtx lock",
> > > "2020-08-13T15:19:03.279Z","7","timer delta: 0",
> > > "2020-08-13T15:19:03.280Z","2","SSL_shutdown() failed (SSL:
> > > error:1409F07F:SSL routines:ssl3_write_pending:bad write retry)
> while
> > > closing request, client: XXX.XXX.XXX.XXX, server:
> > > XXX.XXX.XXX.XXX:443","9140"
> > > "2020-08-13T15:19:03.280Z","7","epoll: fd:322 ev:0005
> > > d:00007F0A0FCDDEB0",
> > > "2020-08-13T15:19:03.280Z","7","epoll: fd:54 ev:0004
> > > d:00007F0A0FCDFAC9",
> > > 
> > > And nothing else in the log saying "SSL_shutdow()", while there 
> > > should be a debug messages like "SSL_shutdown: -1" and 
> > > "SSL_get_error: ..." right before the message, and nothing at all 
> > > related to the connection 9140.
> > > 
> > > It looks like the debug logging is only enabled on the global 
> > > level, but disabled at http or server level.  Please see the part 
> > > starting at "Note that redefining the log without also specifying 
> > > the debug level will disable the debugging log" in the "A 
> > > debugging log" article 
> > > (http://nginx.org/en/docs/debugging_log.html).
> > 
> > So... Bad news: i cannot capture the event when full debug enabled.
> Server
> > cannot handle the load and our service partially down at that time.
> > 
> > What can i say is that this problem reveal itself on all servers
> with new
> > nginx version.
> > 
> > I'll send you the link privately where you can get our
> (some-redacted)
> > config files.
> 
> Thank you for your efforts.  Just in case, it is possible to 
> configure debug logging only for parts of the load - using the 
> debug_connection directive with large networks in CIDR notation 
> (http://nginx.org/r/debug_connection).  It's probably not needed 
> in this particular case, see below.
> 
> I was able to reporoduce an "SSL_shutdown() failed (SSL: ... bad 
> write retry)" error at least in one case, similar to the one 
> previously observed with SSL_write() in 
> https://trac.nginx.org/nginx/ticket/1194.  Previously, this case 
> wasn't causing SSL_shutdown() errors, but SSL shutdown fix 
> introduced in nginx 1.19.2 revealed the problem.
> 
> The following patch should fix this.  It was discussed previously 
> as a possible fix for other SSL_shutdown() errors fixed in 1.19.2, 
> but wasn't commited as there were concerns it will effectively 
> disable SSL shutdown in some unrelated cases where c->error flag 
> is misused.  Now it is more or less clear that the change is 
> needed.
> 
> Patch (it would be great if you'll be able to test if it fixes the 
> problem for you):
> 
> # HG changeset patch
> # User Maxim Dounin <mdounin at mdounin.ru>
> # Date 1597531639 -10800
> #      Sun Aug 16 01:47:19 2020 +0300
> # Node ID be7a3155e00161baf7359ffa73a3a226f1e487c9
> # Parent  7d46c9f56c9afe34a38bb3aea99550a2fd884280
> SSL: disabled shutdown after connection errors.
> 
> This fixes ""SSL_shutdown() failed (SSL: ... bad write retry)" errors
> as observed on the second SSL_shutdown() call after SSL shutdown fixes
> in
> 09fb2135a589 (1.19.2), notably when sending fails in
> ngx_http_test_expect(),
> similarly to ticket 1194.
> 
> Note that there are some places where c->error is misused to prevent
> further output, such as ngx_http_v2_finalize_connection() if there
> are pending streams, or in filter finalization.  These places seem
> to be extreme enough to don't care about missing shutdown though.
> For example, filter finalization currently prevents keepalive from
> being used.
> 
> diff --git a/src/event/ngx_event_openssl.c
> b/src/event/ngx_event_openssl.c
> --- a/src/event/ngx_event_openssl.c
> +++ b/src/event/ngx_event_openssl.c
> @@ -2793,7 +2793,7 @@ ngx_ssl_shutdown(ngx_connection_t *c)
>          return NGX_OK;
>      }
>  
> -    if (c->timedout) {
> +    if (c->timedout || c->error) {
>          mode = SSL_RECEIVED_SHUTDOWN|SSL_SENT_SHUTDOWN;
>          SSL_set_quiet_shutdown(c->ssl->connection, 1);
>  
> 
> 
> 
> -- 
> Maxim Dounin
> http://mdounin.ru/
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

Good day.

Patch mentioned above solved half the problems.

SSL_shutdown() failed (SSL: error:1409F07F:SSL
routines:ssl3_write_pending:bad write retry) while processing HTTP/2
connection

Still remains in the logs.

Regards,
Alexander.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,289087,289130#msg-289130



More information about the nginx mailing list