SSLEngine closed already exception triggered by reload

[Maxim Dounin]

Hello!

On Mon, Nov 05, 2018 at 09:14:33AM -0500, nginxuser2018 wrote:

> I noticed that if I setup a simple scenario where a client is making
> concurrent requests on a server with nginx configured as a reverse proxy and
> SSL traffic termination endpoint, if I trigger a reload with 'nginx -s
> reload' mid requests,  often times the client will throw an
> 'javax.net.ssl.SSLException: SSLEngine closed already
> at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source)' exception.
> 
> I'm using Scala with the Play framework, which uses netty under the hood.
> 
> Is there any configuration that could avoid these exceptions being thrown? 
> 
> I cannot reproduce it using for example using Play to serve HTTPS, so I can
> possibly rule out a problem in the client and think it is a problem with
> nginx?

On Tue, Nov 06, 2018 at 08:49:07AM -0500, nginxuser2018 wrote:

> One setting that I noticed mitigates the problem is to use `lingering_close
> always;` however in our infrastructure this can lead to the build up of
> worker processes (for the duration of the lingering_timeout).  What are the
> advantages and drawbacks of using this setting?

Upon configuration reload, nginx will close connections after it 
finishes processing already active requests in these connections.  
And given that "lingering_close always;" helps, I think there are 
two possible cases here:

1. Closing the connection by nginx happens and the wrong time, 
right before the next request is received on this connection, so 
RST is sent on the connection before the client is able get the 
response and the connection close.  If this is indeed the case,
using "lingering_close always;" might be the right thing to do - 
or, alternatively, lingering close automatic logic might need to 
be improved.

2. The client isn't smart enough to check that the connection is 
closed before sending the next request, and/or isn't smart enough 
to recover from asynchronous close events (a good description can 
be found in RFC 2616, "8.1.4 Practical Considerations", 
https://tools.ietf.org/html/rfc2616#section-8.1.4).  In this case, 
a proper fix would be to improve the client.

-- 
Maxim Dounin
http://mdounin.ru/