Nginx throttling issue?

[John Melom]

Hi Maxim,

I've looked at the nstat data and found the following values for counters:

> nstat -az | grep -I listen
TcpExtListenOverflows           0                  0.0
TcpExtListenDrops               0                  0.0
TcpExtTCPFastOpenListenOverflow 0                  0.0


nstat -az | grep -i retra
TcpRetransSegs                  12157              0.0
TcpExtTCPLostRetransmit         0                  0.0
TcpExtTCPFastRetrans            270                0.0
TcpExtTCPForwardRetrans         11                 0.0
TcpExtTCPSlowStartRetrans       0                  0.0
TcpExtTCPRetransFail            0                  0.0
TcpExtTCPSynRetrans             25                 0.0

Assuming the above "Listen" counters provide data about the overflow issue you mention, then there are no overflows on my system.  While retransmissions are happening, it doesn't seem they are related to listen queue overflows.


Am I looking at the correct data items?  Is my interpretation of the data correct?  If so, do you have any other ideas I could investigate?

Thanks,

John

-----Original Message-----
From: nginx [mailto:nginx-bounces at nginx.org] On Behalf Of John Melom
Sent: Tuesday, March 27, 2018 8:52 AM
To: nginx at nginx.org
Subject: RE: Nginx throttling issue?

Maxim,

Thank you for your reply.  I will look to see if "netstat -s" detects any listen queue overflows.

John


-----Original Message-----
From: nginx [mailto:nginx-bounces at nginx.org] On Behalf Of Maxim Dounin
Sent: Tuesday, March 27, 2018 6:55 AM
To: nginx at nginx.org
Subject: Re: Nginx throttling issue?

Hello!

On Mon, Mar 26, 2018 at 08:21:27PM +0000, John Melom wrote:

> I am load testing our system using Jmeter as a load generator.
> We execute a script consisting of an https request executing in a
> loop.  The loop does not contain a think time, since at this point I
> am not trying to emulate a “real user”.  I want to get a quick look at
> our system capacity.  Load on our system is increased by increasing
> the number of Jmeter threads executing our script.  Each Jmeter thread
> references different data.
>
> Our system is in AWS with an ELB fronting Nginx, which serves as a
> reverse proxy for our Docker Swarm application cluster.
>
> At moderate loads, a subset of our https requests start experiencing
> to a 1 second delay in addition to their normal response time.  The
> delay is not due to resource contention.
> System utilizations remain low.  The response times cluster around 4
> values:  0 millilseconds, 50 milliseconds, 1 second, and 1.050
> seconds.  Right now, I am most interested in understanding and
> eliminating the 1 second delay that gives the clusters at 1 second and
> 1.050 seconds.
>
> The attachment shows a response time scatterplot from one of our runs.
> The x-axis is the number of seconds into the run, the y-axis is the
> response time in milliseconds.  The plotted data shows the response
> time of requests at the time they occurred in the run.
>
> If I run the test bypassing the ELB and Nginx, this delay does not
> occur.
> If I bypass the ELB, but include Nginx in the request path, the delay
> returns.
>
> This leads me to believe the 1 second delay is coming from Nginx.

There are no magic 1 second delays in nginx - unless you've configured something explicitly.

Most likely, the 1 second delay is coming from TCP retransmission timeout during connection establishment due to listen queue overflows.  Check "netstat -s" to see if there are any listen queue overflows on your hosts.

[...]

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
nginx at nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

________________________________
NOTE: This email message and any attachments are for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you have received this e-mail in error, please contact the sender by replying to this email, and destroy all copies of the original message and any material included with this email.
_______________________________________________
nginx mailing list
nginx at nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx

________________________________
NOTE: This email message and any attachments are for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you have received this e-mail in error, please contact the sender by replying to this email, and destroy all copies of the original message and any material included with this email.