Socket read errors/dropped connections during reload

[Maxim Dounin]

Hello!

On Mon, Nov 26, 2018 at 05:58:26AM -0500, dmitry.murashenkov wrote:

> I've been writing a custom load test that performs nginx reload (with same
> actual config) and noticed that sometimes a single connection get's dropped
> during reload. The client was in Java, nginx on localhost in Docker under
> RHEL 7.5 and about 6k req/sec.
> 
> Can somebody comment - if this is expected behavior or possibly
> bug/configuration error?
> 
> I've managed to dump traffic and find that single request that failed (GET
> in packet 14968 fails):
> https://drive.google.com/file/d/1I-orMdoZ-zCTiCBFJszWyba2qsO-c1oz/view?usp=sharing
> 
> The connection has served several requests at this point already and it goes
> like this:
> >GET
> <200
> >GET
> <200
> >GET
> <FIN

Upon configuration reload, nginx closes all idle connections - 
that is, connections which are currently in the keepalive state.  
And much like any close of a keepalive connection, this results in 
a race with the client trying to send the next request in this 
connection.

Quoting section "8.1.4 Practical Considerations" of RFC 2616 
(https://tools.ietf.org/html/rfc2616#section-8.1.4):

   A client, server, or proxy MAY close the transport connection at any
   time. For example, a client might have started to send a new request
   at the same time that the server has decided to close the "idle"
   connection. From the server's point of view, the connection is being
   closed while it was idle, but from the client's point of view, a
   request is in progress.

   This means that clients, servers, and proxies MUST be able to recover
   from asynchronous close events. Client software SHOULD reopen the
   transport connection and retransmit the aborted sequence of requests
   without user interaction so long as the request sequence is
   idempotent (see section 9.1.2). Non-idempotent methods or sequences
   MUST NOT be automatically retried, although user agents MAY offer a
   human operator the choice of retrying the request(s). Confirmation by
   user-agent software with semantic understanding of the application
   MAY substitute for user confirmation. The automatic retry SHOULD NOT
   be repeated if the second sequence of requests fails.

-- 
Maxim Dounin
http://mdounin.ru/