Nginx throttling issue?

[Maxim Dounin]

Hello!

On Mon, Mar 26, 2018 at 08:21:27PM +0000, John Melom wrote:

> I am load testing our system using Jmeter as a load generator.  
> We execute a script consisting of an https request executing in 
> a loop.  The loop does not contain a think time, since at this 
> point I am not trying to emulate a “real user”.  I want to get a 
> quick look at our system capacity.  Load on our system is 
> increased by increasing the number of Jmeter threads executing 
> our script.  Each Jmeter thread references different data.
> 
> Our system is in AWS with an ELB fronting Nginx, which serves as 
> a reverse proxy for our Docker Swarm application cluster.
> 
> At moderate loads, a subset of our https requests start 
> experiencing to a 1 second delay in addition to their normal 
> response time.  The delay is not due to resource contention.  
> System utilizations remain low.  The response times cluster 
> around 4 values:  0 millilseconds, 50 milliseconds, 1 second, 
> and 1.050 seconds.  Right now, I am most interested in 
> understanding and eliminating the 1 second delay that gives the 
> clusters at 1 second and 1.050 seconds.
> 
> The attachment shows a response time scatterplot from one of our 
> runs.  The x-axis is the number of seconds into the run, the 
> y-axis is the response time in milliseconds.  The plotted data 
> shows the response time of requests at the time they occurred in 
> the run.
> 
> If I run the test bypassing the ELB and Nginx, this delay does 
> not occur.
> If I bypass the ELB, but include Nginx in the request path, the 
> delay returns.
> 
> This leads me to believe the 1 second delay is coming from 
> Nginx.

There are no magic 1 second delays in nginx - unless you've 
configured something explicitly.

Most likely, the 1 second delay is coming from TCP retransmission 
timeout during connection establishment due to listen queue 
overflows.  Check "netstat -s" to see if there are any listen 
queue overflows on your hosts.

[...]

-- 
Maxim Dounin
http://mdounin.ru/