<div dir="ltr">What language is your upstream API written in and are you hosting in Amazon? Even if you aren't hosting in Amazon, it seems your values are quite low for connect, read and send.<br><br>If you are running NodeJS for the upstream, one thing I have found is that certain values need to be offset to avoid the 502 errors:<div><br></div><div>1. nGinx: <br>proxy_read_timeout and proxy_send_timeout should be set higher if not equal (60s)<br><br>2. NodeJS: <br><div style="color:rgb(197,200,198);background-color:rgb(30,30,30);font-family:Consolas,"Courier New",monospace;font-size:13px;line-height:18px;white-space:pre"><div><span style="color:rgb(96,137,180)">server</span>.<span style="color:rgb(152,114,162)">keepAliveTimeout</span> <span style="color:rgb(103,104,103)">=</span> <span style="color:rgb(96,137,180)">70</span> <span style="color:rgb(103,104,103)">*</span> <span style="color:rgb(96,137,180)">1000</span>;</div><div><span style="color:rgb(96,137,180)">server</span>.<span style="color:rgb(152,114,162)">headersTimeout</span> <span style="color:rgb(103,104,103)">=</span> <span style="color:rgb(96,137,180)">75</span> <span style="color:rgb(103,104,103)">*</span> <span style="color:rgb(96,137,180)">1000</span>;</div></div><div><br>keepaliveTimeout for the NodeJS app needs to be longer than the nginx timeouts and headersTimeout needs to slightly extend keepAliveTimeout.<br><br>Not sure if this applies to other languages but that was an issue we ran into a few years ago.<br><br></div><div>Craig</div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 25, 2024 at 8:32 AM Sébastien Rebecchi <<a href="mailto:srebecchi@kameleoon.com">srebecchi@kameleoon.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="padding:0px 0px 0px 2px"><div style="color:rgb(0,0,0);font-family:monospace;font-size:10pt"><p style="margin:0px">Hello</p><p style="margin:0px"><br></p><p style="margin:0px">I have an issue with <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">nginx</span> closing prematurely connections when reload is performed.</p><p style="margin:0px"><br></p><p style="margin:0px">I have some <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">nginx</span> servers configured to proxy_pass requests to an upstream group. This group itself is composed of several servers which are <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">nginx</span> themselves, and is configured to use <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">keepalive</span> connections.</p><p style="margin:0px">When I trigger a reload (-s reload) on an <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">nginx</span> of one of the servers which is target of the upstream, I see in error logs of all servers in front that connection was reset by the <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">nginx</span> which was reloaded.</p><p style="margin:0px"><br></p><p style="margin:0px">Here configuration of upstream group (IPs are hidden replaced by IP_X):</p><p style="margin:0px">--- BEGIN ---</p><p style="margin:0px">upstream data_api {</p><p style="margin:0px"> random;</p><p style="margin:0px"><br></p><p style="margin:0px"> server IP_1:80 max_fails=3 fail_timeout=30s;</p><p style="margin:0px"> server IP_2:80 max_fails=3 fail_timeout=30s;</p><p style="margin:0px"> server IP_3:80 max_fails=3 fail_timeout=30s;</p><p style="margin:0px"> server IP_4:80 max_fails=3 fail_timeout=30s;</p><p style="margin:0px"> server IP_5:80 max_fails=3 fail_timeout=30s;</p><p style="margin:0px"> server IP_6:80 max_fails=3 fail_timeout=30s;</p><p style="margin:0px"> server IP_7:80 max_fails=3 fail_timeout=30s;</p><p style="margin:0px"> server IP_8:80 max_fails=3 fail_timeout=30s;</p><p style="margin:0px"> server IP_9:80 max_fails=3 fail_timeout=30s;</p><p style="margin:0px"> server IP_10:80 max_fails=3 fail_timeout=30s;</p><p style="margin:0px"><br></p><p style="margin:0px"> <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">keepalive</span> 20;</p><p style="margin:0px">}</p><p style="margin:0px">--- END ---</p><p style="margin:0px"><br></p><p style="margin:0px">Here configuration of the location using this upstream:</p><p style="margin:0px">--- BEGIN ---</p><p style="margin:0px">location / {</p><p style="margin:0px"> proxy_pass <a href="http://data_api" target="_blank">http://data_api</a>;</p><p style="margin:0px"><br></p><p style="margin:0px"> proxy_http_version 1.1;</p><p style="margin:0px"> proxy_set_header Connection "";</p><p style="margin:0px"><br></p><p style="margin:0px"> proxy_set_header Host $host;</p><p style="margin:0px"> proxy_set_header X-Real-IP $real_ip;</p><p style="margin:0px"> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;</p><p style="margin:0px"><br></p><p style="margin:0px"> proxy_connect_timeout 2s;</p><p style="margin:0px"> proxy_send_timeout 6s;</p><p style="margin:0px"> proxy_read_timeout 10s;</p><p style="margin:0px"><br></p><p style="margin:0px"> proxy_next_upstream error timeout <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">http</span>_502 <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">http</span>_504;</p><p style="margin:0px">}</p><p style="margin:0px">--- END ---</p><p style="margin:0px"><br></p><p style="margin:0px">And here the kind of error messages I get when I reload <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">nginx</span> of "IP_1":</p><p style="margin:0px">--- BEGIN ---</p><p style="margin:0px">2024/03/25 11:24:25 [error] 3758170#0: *1795895162 <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">recv</span>() failed (104: Connection reset by peer) while reading response header from upstream, client: CLIENT_IP_HIDDEN, server: SERVER_HIDDEN, request: "POST /REQUEST_LOCATION_HIDDEN HTTP/2.0", upstream: "<a href="http://IP_1:80/REQUEST_LOCATION_HIDDEN" target="_blank">http://IP_1:80/REQUEST_LOCATION_HIDDEN</a>", host: "HOST_HIDDEN", referrer: "REFERRER_HIDDEN"</p><p style="margin:0px">--- END ---</p><p style="margin:0px"><br></p><p style="margin:0px">I thought -s reload was doing graceful shutdown of connections. Is it due to the fact that <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">nginx</span> can not handle that when using <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">keepalive</span> connections? Is it a bug?</p><p style="margin:0px">I am using <span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">nginx</span> 1.24.0 everywhere, no particular </p><p style="margin:0px"><br></p><p style="margin:0px">Thank you for any help.</p><p style="margin:0px"><br></p><p style="margin:0px"><span style="text-decoration-line:underline;text-decoration-style:wavy;text-decoration-color:rgb(255,128,64)">Sébastien</span></p><p style="margin:0px"></p></div></div></div>
_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a><br>
<a href="https://mailman.nginx.org/mailman/listinfo/nginx" rel="noreferrer" target="_blank">https://mailman.nginx.org/mailman/listinfo/nginx</a><br>
</blockquote></div>