<div dir="ltr">Hi Igor,<div><br></div><div>I did not have error_log directive at main context, so it took default conf, which seems why i got only 1 log file. I added directive and now I have more logs when I do nginx -s reload:</div>2024/03/29 09:04:20 [notice] 1064394#0: signal process started<br>2024/03/29 09:04:20 [notice] 3718160#0: signal 1 (SIGHUP) received from 1064394, reconfiguring<br>2024/03/29 09:04:20 [notice] 3718160#0: reconfiguring<br>2024/03/29 09:04:20 [notice] 3718160#0: using the "epoll" event method<br>2024/03/29 09:04:20 [notice] 3718160#0: start worker processes<br>2024/03/29 09:04:20 [notice] 3718160#0: start worker process 1064395<br>2024/03/29 09:04:20 [notice] 3718160#0: start worker process 1064396<br>2024/03/29 09:04:20 [notice] 3718160#0: start worker process 1064397<br>2024/03/29 09:04:20 [notice] 3718160#0: start worker process 1064398<br>2024/03/29 09:04:20 [notice] 3718160#0: start worker process 1064399<br>2024/03/29 09:04:20 [notice] 3718160#0: start worker process 1064400<br>2024/03/29 09:04:20 [notice] 3718160#0: start worker process 1064401<br>2024/03/29 09:04:20 [notice] 3718160#0: start worker process 1064402<br>2024/03/29 09:04:20 [notice] 3718160#0: start worker process 1064403<br>2024/03/29 09:04:20 [notice] 3718160#0: start worker process 1064404<br>2024/03/29 09:04:20 [notice] 3718160#0: start worker process 1064405<br>2024/03/29 09:04:20 [notice] 3718160#0: start worker process 1064406<br>2024/03/29 09:04:20 [notice] 1063598#0: gracefully shutting down<br>2024/03/29 09:04:20 [notice] 1063599#0: gracefully shutting down<br>2024/03/29 09:04:20 [notice] 1063600#0: gracefully shutting down<br>2024/03/29 09:04:20 [notice] 1063601#0: gracefully shutting down<br>2024/03/29 09:04:20 [notice] 1063602#0: gracefully shutting down<br>2024/03/29 09:04:20 [notice] 1063603#0: gracefully shutting down<br>2024/03/29 09:04:20 [notice] 1063604#0: gracefully shutting down<br>2024/03/29 09:04:20 [notice] 1063607#0: gracefully shutting down<br>2024/03/29 09:04:20 [notice] 1063608#0: gracefully shutting down<br>2024/03/29 09:04:20 [notice] 1063597#0: gracefully shutting down<br>2024/03/29 09:04:20 [notice] 1063605#0: gracefully shutting down<br>2024/03/29 09:04:20 [notice] 1063609#0: gracefully shutting down<br>2024/03/29 09:04:23 [notice] 3718160#0: signal 17 (SIGCHLD) received from 3989432<br>2024/03/29 09:04:23 [notice] 3718160#0: worker process 3989432 exited with code 0<br>2024/03/29 09:04:23 [notice] 3718160#0: signal 29 (SIGIO) received<br>2024/03/29 09:04:26 [notice] 1060347#0: exiting<br>2024/03/29 09:04:26 [notice] 1060347#0: exit<br>2024/03/29 09:04:26 [notice] 3718160#0: signal 17 (SIGCHLD) received from 1060347<br>2024/03/29 09:04:26 [notice] 3718160#0: worker process 1060347 exited with code 0<br>2024/03/29 09:04:26 [notice] 3718160#0: signal 29 (SIGIO) received<br>2024/03/29 09:04:29 [notice] 3718160#0: signal 17 (SIGCHLD) received from 3989423<br>2024/03/29 09:04:29 [notice] 3718160#0: worker process 3989423 exited with code 0<br><div>2024/03/29 09:04:29 [notice] 3718160#0: signal 29 (SIGIO) received</div><div>... etc ...</div><div><br></div><div>Could the pb I encounter be linked to that discussion? <a href="https://mailman.nginx.org/pipermail/nginx-devel/2024-January/YSJATQMPXDIBETCDS46OTKUZNOJK6Q22.html" target="_blank">https://mailman.nginx.org/pipermail/nginx-devel/2024-January/YSJATQMPXDIBETCDS46OTKUZNOJK6Q22.html</a></div><div>Seems a race condition between a client that have started to send a new request at the same time that the server has decided to close the idle connection. Is there a plan to add a bugfix in nginx to handle that properly?</div><div><br></div><div>Thanks,</div><div><br></div><div>Sébastien</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le ven. 29 mars 2024 à 00:04, Igor Ippolitov <<a href="mailto:iippolitov@nginx.com" target="_blank">iippolitov@nginx.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><u></u>
<div>
<div>Sébastien,</div>
<div><br>
</div>
<div>Is it possible that messages go to
another log file? These messages go to the main error log file,
defined in the root context.</div>
<div>Another common pitfall is a log level
above notice. Try setting error log to a more verbose one, maybe?<br>
</div>
<div><br>
</div>
<div>Regards,<br>
Igor.<br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>On 28/03/2024 18:27, Sébastien Rebecchi
wrote:<br>
</div>
<blockquote type="cite">
<div dir="auto">Hi Igor,
<div dir="auto"><br>
</div>
<div dir="auto">Thanks for the answer.</div>
<div dir="auto"><br>
</div>
<div dir="auto">I really got that message 'signal process
started' every time i do 'nginx -s reload' and this is the
only log line I have, I don't have the other lines you
mentioned. Is there anything to do to enable those logs?</div>
<div dir="auto"><br>
</div>
Sébastien<br>
<br>
<div class="gmail_quote" dir="auto">
<div dir="ltr" class="gmail_attr">Le jeu. 28 mars 2024, 16:40,
Igor Ippolitov <<a href="mailto:iippolitov@nginx.com" target="_blank">iippolitov@nginx.com</a>>
a écrit :<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Sébastien,</div>
<div><br>
</div>
<div>The message about the signal process is only the
beginning of the process.</div>
<div>You are interested in messages like the following:<br>
<br>
<span style="font-family:monospace">
<blockquote type="cite"><span style="font-family:monospace"><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)">2024/03/26
13:36:36 [notice] 723#723: signal 1 (SIGHUP)
received from 69064, reconfiguring </span><br>
2024/03/26 13:36:36 [notice] 723#723:
reconfiguring <br>
2024/03/26 13:36:36 [notice] 723#723: using the
"epoll" event method <br>
2024/03/26 13:36:36 [notice] 723#723: start worker
processes <br>
2024/03/26 13:36:36 [notice] 723#723: start worker
process 69065 <br>
2024/03/26 13:36:36 [notice] 723#723: start worker
process 69066 <br>
2024/03/26 13:36:36 [notice] 723#723: start cache
manager process 69067 <br>
2024/03/26 13:36:36 [notice] 61903#61903:
gracefully shutting down <br>
2024/03/26 13:36:36 [notice] 61905#61905: exiting
<br>
2024/03/26 13:36:36 [notice] 61903#61903: exiting
<br>
2024/03/26 13:36:36 [notice] 61904#61904:
gracefully shutting down <br>
2024/03/26 13:36:36 [notice] 61904#61904: exiting
<br>
2024/03/26 13:36:36 [notice] 61903#61903: exit</span></blockquote>
<br>
</span></div>
<div>Note the 'gracefully shutting down' and 'exiting'
message from workers. Also the 'start' and
'reconfiguring' messages from the master process.</div>
<div>There should be a similar sequence somewhere in your
logs.</div>
<div>Having these logs may help explaining what happens on
a reload.<br>
</div>
<div><br>
</div>
<div>Kind regards,</div>
<div>Igor.<br>
</div>
<div><br>
</div>
<div>On 26/03/2024 12:41, Sébastien Rebecchi wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi Igor
<div><br>
</div>
<div>There is no special logs on the IP_1 (the
reloaded one) side, only 1 log line, which is
expected:</div>
<div>--- BEGIN ---<br>
</div>
<div>2024/03/26 13:37:55 [notice] 3928855#0: signal
process started<br>
</div>
<div>--- END ---<br>
</div>
<div><br>
</div>
<div>I did not configure worker_shutdown_timeout, it
is unlimited.</div>
<div><br>
</div>
<div>Sébastien.</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">Le lun. 25 mars 2024
à 17:59, Igor Ippolitov <<a href="mailto:iippolitov@nginx.com" rel="noreferrer" target="_blank">iippolitov@nginx.com</a>>
a écrit :<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Sébastien,</div>
<div><br>
</div>
<div>Nginx should keep active connections open and
wait for a request to complete before closing.<br>
</div>
<div>A reload starts a new set of workers while
old workers wait for old connections to shut
down.</div>
<div>The only exception I'm aware of is having
worker_shutdown_timeout configured: in this case
a worker will wait till this timeout and
forcibly close a connection. Be default there is
no timeout.<br>
</div>
<div><br>
</div>
<div>It would be curious to see error log of nginx
at IP_1 (the reloaded one) while the reload
happens. It may explain the reason for
connection resets.<br>
</div>
<div><br>
</div>
<div>Kind regards,</div>
<div>Igor.<br>
</div>
<div><br>
</div>
<div>On 25/03/2024 12:31, Sébastien Rebecchi
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div style="padding:0px 0px 0px 2px">
<div style="color:rgb(0,0,0);font-family:monospace;font-size:10pt">
<p style="margin:0px">Hello</p>
<p style="margin:0px"><br>
</p>
<p style="margin:0px">I have an issue with
<span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">nginx</span>
closing prematurely connections when
reload is performed.</p>
<p style="margin:0px"><br>
</p>
<p style="margin:0px">I have some <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">nginx</span>
servers configured to proxy_pass
requests to an upstream group. This
group itself is composed of several
servers which are <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">nginx</span>
themselves, and is configured to use <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">keepalive</span>
connections.</p>
<p style="margin:0px">When I trigger a
reload (-s reload) on an <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">nginx</span>
of one of the servers which is target of
the upstream, I see in error logs of all
servers in front that connection was
reset by the <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">nginx</span>
which was reloaded.</p>
<p style="margin:0px"><br>
</p>
<p style="margin:0px">Here configuration
of upstream group (IPs are hidden
replaced by IP_X):</p>
<p style="margin:0px">--- BEGIN ---</p>
<p style="margin:0px">upstream data_api {</p>
<p style="margin:0px"> random;</p>
<p style="margin:0px"><br>
</p>
<p style="margin:0px"> server IP_1:80
max_fails=3 fail_timeout=30s;</p>
<p style="margin:0px"> server IP_2:80
max_fails=3 fail_timeout=30s;</p>
<p style="margin:0px"> server IP_3:80
max_fails=3 fail_timeout=30s;</p>
<p style="margin:0px"> server IP_4:80
max_fails=3 fail_timeout=30s;</p>
<p style="margin:0px"> server IP_5:80
max_fails=3 fail_timeout=30s;</p>
<p style="margin:0px"> server IP_6:80
max_fails=3 fail_timeout=30s;</p>
<p style="margin:0px"> server IP_7:80
max_fails=3 fail_timeout=30s;</p>
<p style="margin:0px"> server IP_8:80
max_fails=3 fail_timeout=30s;</p>
<p style="margin:0px"> server IP_9:80
max_fails=3 fail_timeout=30s;</p>
<p style="margin:0px"> server IP_10:80
max_fails=3 fail_timeout=30s;</p>
<p style="margin:0px"><br>
</p>
<p style="margin:0px"> <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">keepalive</span>
20;</p>
<p style="margin:0px">}</p>
<p style="margin:0px">--- END ---</p>
<p style="margin:0px"><br>
</p>
<p style="margin:0px">Here configuration
of the location using this upstream:</p>
<p style="margin:0px">--- BEGIN ---</p>
<p style="margin:0px">location / {</p>
<p style="margin:0px"> proxy_pass <a href="http://data_api" rel="noreferrer" target="_blank">http://data_api</a>;</p>
<p style="margin:0px"><br>
</p>
<p style="margin:0px"> proxy_http_version
1.1;</p>
<p style="margin:0px"> proxy_set_header
Connection "";</p>
<p style="margin:0px"><br>
</p>
<p style="margin:0px"> proxy_set_header
Host $host;</p>
<p style="margin:0px"> proxy_set_header
X-Real-IP $real_ip;</p>
<p style="margin:0px"> proxy_set_header
X-Forwarded-For
$proxy_add_x_forwarded_for;</p>
<p style="margin:0px"><br>
</p>
<p style="margin:0px">
proxy_connect_timeout 2s;</p>
<p style="margin:0px"> proxy_send_timeout
6s;</p>
<p style="margin:0px"> proxy_read_timeout
10s;</p>
<p style="margin:0px"><br>
</p>
<p style="margin:0px"> proxy_next_upstream
error timeout <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">http</span>_502
<span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">http</span>_504;</p>
<p style="margin:0px">}</p>
<p style="margin:0px">--- END ---</p>
<p style="margin:0px"><br>
</p>
<p style="margin:0px">And here the kind of
error messages I get when I reload <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">nginx</span>
of "IP_1":</p>
<p style="margin:0px">--- BEGIN ---</p>
<p style="margin:0px">2024/03/25 11:24:25
[error] 3758170#0: *1795895162 <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">recv</span>()
failed (104: Connection reset by peer)
while reading response header from
upstream, client: CLIENT_IP_HIDDEN,
server: SERVER_HIDDEN, request: "POST
/REQUEST_LOCATION_HIDDEN HTTP/2.0",
upstream: "<a href="http://IP_1:80/REQUEST_LOCATION_HIDDEN" rel="noreferrer" target="_blank">http://IP_1:80/REQUEST_LOCATION_HIDDEN</a>",
host: "HOST_HIDDEN", referrer:
"REFERRER_HIDDEN"</p>
<p style="margin:0px">--- END ---</p>
<p style="margin:0px"><br>
</p>
<p style="margin:0px">I thought -s reload
was doing graceful shutdown of
connections. Is it due to the fact that
<span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">nginx</span>
can not handle that when using <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">keepalive</span>
connections? Is it a bug?</p>
<p style="margin:0px">I am using <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">nginx</span>
1.24.0 everywhere, no particular </p>
<p style="margin:0px"><br>
</p>
<p style="margin:0px">Thank you for any
help.</p>
<p style="margin:0px"><br>
</p>
<p style="margin:0px"><span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">Sébastien</span></p>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
nginx mailing list
<a href="mailto:nginx@nginx.org" rel="noreferrer" target="_blank">nginx@nginx.org</a>
<a href="https://mailman.nginx.org/mailman/listinfo/nginx" rel="noreferrer" target="_blank">https://mailman.nginx.org/mailman/listinfo/nginx</a>
</pre>
</blockquote>
<p><br>
</p>
</div>
</blockquote>
</div>
</blockquote>
<p><br>
</p>
</div>
</blockquote>
</div>
</div>
</blockquote>
<p><br>
</p>
</div>
</blockquote></div>