Nginx throttling issue?

Fri Apr 6 17:11:36 UTC 2018

 Even though it shouldn't be reaching your limits, limit_req does delay in
1 second increments which sounds like it could be responsible for this. You
should see error log entries if this happens (severity warning). Have you
tried without the limit_req option? You can also use the nodelay option to
avoid the delaying behavior.

http://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req

On Thu, Apr 5, 2018 at 6:45 AM, Peter Booth <peter_booth at me.com> wrote:

> John,
>
> I think that you need to understand what is happening on your host
> throughout the duration of the test. Specifically, what is happening with
> the tcp connections. If you run netstat and grep for tcp and do this in a
> loop every say five seconds then you’ll see how many connections peak get
> created.
> If the thing you are testing exists in production then you are lucky. You
> can do the same in production and see what it is that you need to replicate.
>
> You didn’t mention whether you had persistent connections (http keep
> alive) configured. This is key to maximizing scalability. You did say that
> you were using SSL. If it were me I’d use a load generator that more
> closely resembles the behavior of real users on a website. Wrk2, Tsung,
> httperf, Gatling are examples of some that do. Using jmeter with zero think
> time is a very common anti pattern that doesn’t behave anything like real
> users. I think of it as the lazy performance tester pattern.
>
> Imagine a real web server under heavy load from human beings. You will see
> thousands of concurrent connections but fewer concurrent requests in
> flight. With the jmeter zero think time model then you are either creating
> new connections or reusing them - so either you have a shitload of
> connections and your nginx process starts running out of file handles or
> you are jamming requests down a single connection- neither of which
> resemble reality.
>
> If you are committed to using jmeter for some reason then use more
> instances with real thinktimes. Each instance’s connection wil have a
> different source port
>
> Sent from my iPhone
>
> > On Apr 4, 2018, at 5:20 PM, John Melom <John.Melom at spok.com> wrote:
> >
> > Hi Maxim,
> >
> > I've looked at the nstat data and found the following values for
> counters:
> >
> >> nstat -az | grep -I listen
> > TcpExtListenOverflows           0                  0.0
> > TcpExtListenDrops               0                  0.0
> > TcpExtTCPFastOpenListenOverflow 0                  0.0
> >
> >
> > nstat -az | grep -i retra
> > TcpRetransSegs                  12157              0.0
> > TcpExtTCPLostRetransmit         0                  0.0
> > TcpExtTCPFastRetrans            270                0.0
> > TcpExtTCPForwardRetrans         11                 0.0
> > TcpExtTCPSlowStartRetrans       0                  0.0
> > TcpExtTCPRetransFail            0                  0.0
> > TcpExtTCPSynRetrans             25                 0.0
> >
> > Assuming the above "Listen" counters provide data about the overflow
> issue you mention, then there are no overflows on my system.  While
> retransmissions are happening, it doesn't seem they are related to listen
> queue overflows.
> >
> >
> > Am I looking at the correct data items?  Is my interpretation of the
> data correct?  If so, do you have any other ideas I could investigate?
> >
> > Thanks,
> >
> > John
> >
> > -----Original Message-----
> > From: nginx [mailto:nginx-bounces at nginx.org] On Behalf Of John Melom
> > Sent: Tuesday, March 27, 2018 8:52 AM
> > To: nginx at nginx.org
> > Subject: RE: Nginx throttling issue?
> >
> > Maxim,
> >
> > Thank you for your reply.  I will look to see if "netstat -s" detects
> any listen queue overflows.
> >
> > John
> >
> >
> > -----Original Message-----
> > From: nginx [mailto:nginx-bounces at nginx.org] On Behalf Of Maxim Dounin
> > Sent: Tuesday, March 27, 2018 6:55 AM
> > To: nginx at nginx.org
> > Subject: Re: Nginx throttling issue?
> >
> > Hello!
> >
> >> On Mon, Mar 26, 2018 at 08:21:27PM +0000, John Melom wrote:
> >>
> >> I am load testing our system using Jmeter as a load generator.
> >> We execute a script consisting of an https request executing in a
> >> loop.  The loop does not contain a think time, since at this point I
> >> am not trying to emulate a “real user”.  I want to get a quick look at
> >> our system capacity.  Load on our system is increased by increasing
> >> the number of Jmeter threads executing our script.  Each Jmeter thread
> >> references different data.
> >>
> >> Our system is in AWS with an ELB fronting Nginx, which serves as a
> >> reverse proxy for our Docker Swarm application cluster.
> >>
> >> At moderate loads, a subset of our https requests start experiencing
> >> to a 1 second delay in addition to their normal response time.  The
> >> delay is not due to resource contention.
> >> System utilizations remain low.  The response times cluster around 4
> >> values:  0 millilseconds, 50 milliseconds, 1 second, and 1.050
> >> seconds.  Right now, I am most interested in understanding and
> >> eliminating the 1 second delay that gives the clusters at 1 second and
> >> 1.050 seconds.
> >>
> >> The attachment shows a response time scatterplot from one of our runs.
> >> The x-axis is the number of seconds into the run, the y-axis is the
> >> response time in milliseconds.  The plotted data shows the response
> >> time of requests at the time they occurred in the run.
> >>
> >> If I run the test bypassing the ELB and Nginx, this delay does not
> >> occur.
> >> If I bypass the ELB, but include Nginx in the request path, the delay
> >> returns.
> >>
> >> This leads me to believe the 1 second delay is coming from Nginx.
> >
> > There are no magic 1 second delays in nginx - unless you've configured
> something explicitly.
> >
> > Most likely, the 1 second delay is coming from TCP retransmission
> timeout during connection establishment due to listen queue overflows.
> Check "netstat -s" to see if there are any listen queue overflows on your
> hosts.
> >
> > [...]
> >
> > --
> > Maxim Dounin
> > http://mdounin.ru/
> > _______________________________________________
> > nginx mailing list
> > nginx at nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx
> >
> > ________________________________
> > NOTE: This email message and any attachments are for the sole use of the
> intended recipient(s) and may contain confidential and/or privileged
> information. Any unauthorized review, use, disclosure or distribution is
> prohibited. If you have received this e-mail in error, please contact the
> sender by replying to this email, and destroy all copies of the original
> message and any material included with this email.
> > _______________________________________________
> > nginx mailing list
> > nginx at nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx
> >
> > ________________________________
> > NOTE: This email message and any attachments are for the sole use of the
> intended recipient(s) and may contain confidential and/or privileged
> information. Any unauthorized review, use, disclosure or distribution is
> prohibited. If you have received this e-mail in error, please contact the
> sender by replying to this email, and destroy all copies of the original
> message and any material included with this email.
> > _______________________________________________
> > nginx mailing list
> > nginx at nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20180406/f1fa27c2/attachment.html>