Nginx prematurely closing connections when reloaded

[Sébastien Rebecchi]

Hello


I have an issue with nginx closing prematurely connections when reload is
performed.


I have some nginx servers configured to proxy_pass requests to an upstream
group. This group itself is composed of several servers which are nginx
themselves, and is configured to use keepalive connections.

When I trigger a reload (-s reload) on an nginx of one of the servers which
is target of the upstream, I see in error logs of all servers in front that
connection was reset by the nginx which was reloaded.


Here configuration of upstream group (IPs are hidden replaced by IP_X):

--- BEGIN ---

upstream data_api {

random;


server IP_1:80 max_fails=3 fail_timeout=30s;

server IP_2:80 max_fails=3 fail_timeout=30s;

server IP_3:80 max_fails=3 fail_timeout=30s;

server IP_4:80 max_fails=3 fail_timeout=30s;

server IP_5:80 max_fails=3 fail_timeout=30s;

server IP_6:80 max_fails=3 fail_timeout=30s;

server IP_7:80 max_fails=3 fail_timeout=30s;

server IP_8:80 max_fails=3 fail_timeout=30s;

server IP_9:80 max_fails=3 fail_timeout=30s;

server IP_10:80 max_fails=3 fail_timeout=30s;


keepalive 20;

}

--- END ---


Here configuration of the location using this upstream:

--- BEGIN ---

location / {

proxy_pass http://data_api;


proxy_http_version 1.1;

proxy_set_header Connection "";


proxy_set_header Host $host;

proxy_set_header X-Real-IP $real_ip;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;


proxy_connect_timeout 2s;

proxy_send_timeout 6s;

proxy_read_timeout 10s;


proxy_next_upstream error timeout http_502 http_504;

}

--- END ---


And here the kind of error messages I get when I reload nginx of "IP_1":

--- BEGIN ---

2024/03/25 11:24:25 [error] 3758170#0: *1795895162 recv() failed (104:
Connection reset by peer) while reading response header from upstream,
client: CLIENT_IP_HIDDEN, server: SERVER_HIDDEN, request: "POST
/REQUEST_LOCATION_HIDDEN HTTP/2.0", upstream: "
http://IP_1:80/REQUEST_LOCATION_HIDDEN", host: "HOST_HIDDEN", referrer:
"REFERRER_HIDDEN"

--- END ---


I thought -s reload was doing graceful shutdown of connections. Is it due
to the fact that nginx can not handle that when using keepalive
connections? Is it a bug?

I am using nginx 1.24.0 everywhere, no particular


Thank you for any help.


Sébastien
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20240325/25d4b4bb/attachment.htm>