<div dir="ltr">Maxim -<div><br></div><div> You were pretty much entirely correct here - although it was actually the firewall (which sits logically between the reverse proxies and the upstreams) which wasn't removing state from the connection table quickly enough. Following the advice here <a href="https://www.nginx.com/blog/overcoming-ephemeral-port-exhaustion-nginx-plus/">https://www.nginx.com/blog/overcoming-ephemeral-port-exhaustion-nginx-plus/</a> resolved the issue for now.</div><div><br></div><div> The suggestion to configure the upstreams to send RST on tcp queue overflows is a really helpful one.</div><div><br></div><div>Thanks again for the help and guidance here.</div><div>Jordan</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Oct 30, 2020 at 4:01 PM Maxim Dounin <<a href="mailto:mdounin@mdounin.ru">mdounin@mdounin.ru</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello!<br>
<br>
On Thu, Oct 29, 2020 at 01:02:57PM -0500, Jordan von Kluck wrote:<br>
<br>
> I am hoping someone on the community list can help steer me in the right<br>
> direction for troubleshooting the following scenario:<br>
> <br>
> I am running a cluster of 4 virtualized nginx open source 1.16.0 servers<br>
> with 4 vCPU cores and 4 GB of RAM each. They serve HTTP (REST API) requests<br>
> to a pool of about 40 different upstream clusters, which range from 2 to 8<br>
> servers within each upstream definition. The upstream application servers<br>
> themselves have multiple workers per server.<br>
> <br>
> I've recently started seeing an issue where the reported response_time and<br>
> typically the reported upstream_response_time the nginx access log are<br>
> drastically different from the reported response on the application servers<br>
> themselves. For example, on some requests the typical average response_time<br>
> would be around 5ms with an upstream_response_time of 4ms. During these<br>
> transient periods of high load (approximately 1200 -1400 rps), the reported<br>
> nginx response_time and upstream_response_time spike up to somewhere around<br>
> 1 second, while the application logs on the upstream servers are still<br>
> reporting the same 4ms response time.<br>
> <br>
> The upstream definitions are very simple and look like:<br>
> upstream rest-api-xyz {<br>
> least_conn;<br>
> server <a href="http://10.1.1.33:8080" rel="noreferrer" target="_blank">10.1.1.33:8080</a> max_fails=3 fail_timeout=30; #<br>
> production-rest-api-xyz01<br>
> server <a href="http://10.1.1.34:8080" rel="noreferrer" target="_blank">10.1.1.34:8080</a> max_fails=3 fail_timeout=30; #<br>
> production-rest-api-xyz02<br>
> }<br>
> <br>
> One avenue that I've considered but does not seem to be the case from the<br>
> instrumentation on the app servers is that they're accepting the requests<br>
> and queueing them in a TCP socket locally. However, running a packet<br>
> capture on both the nginx server and the app server actually shows the http<br>
> request leaving nginx at the end of the time window. I have not looked at<br>
> this down to the TCP handshake to see if the actual negotiation is taking<br>
> an excessive amount of time. I can produce this queueing scenario<br>
> artificially, but it does not appear to be what's happening in my<br>
> production environment in the scenario described above.<br>
> <br>
> Does anyone here have any experience sorting out something like this? The<br>
> upstream_connect_time is not part of the log currently, but if that number<br>
> was reporting high, I'm not entirely sure what would cause that. Similarly,<br>
> if the upstream_connect_time does not account for most of the delay, is<br>
> there anything else I should be looking at?<br>
<br>
Spikes to 1 second suggests that this might be SYN retransmit <br>
timeouts.<br>
<br>
Most likely, this is what happens: your backend cannot cope with <br>
load, so listen queue on the backend overflows. Default behaviour <br>
on most Linux boxes is to drop SYN packets on listen queue <br>
overflows (tcp.ipv4.abort_on_overflow=0). Dropped SYN packets <br>
eventually - after an initial RTO, initial retransmission timeout, <br>
which is 1s on modern Linux systems - result in retransmission and <br>
connection being finally established, but with 1s delay.<br>
<br>
Consider looking at network stats to see if there are actual <br>
listen queue overflows on your backends, something like "nstat -az <br>
TcpExtListenDrops" should be handy. You can also use "ss -nlt" to <br>
see listen queue sizes in real time.<br>
<br>
In many cases such occasional queue overflows under load simply <br>
mean that listen queue size is too low, so minor load fluctuations <br>
might occasionally result in overflows. In this case, using a <br>
larger listen queue might help.<br>
<br>
Also, if the backend servers in question are solely the backend <br>
ones, and there are multiple load-balanced servers as your <br>
configuration suggests, it might be a good idea to configure these <br>
servers to send RST on listen queue overflows, that is, to set <br>
tcp.ipv4.abort_on_overflow to 1. This way nginx will immediately <br>
know that the backend'd listen queue is full and will be able to <br>
try the next upstream server instead.<br>
<br>
-- <br>
Maxim Dounin<br>
<a href="http://mdounin.ru/" rel="noreferrer" target="_blank">http://mdounin.ru/</a><br>
_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx" rel="noreferrer" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a><br>
</blockquote></div>