<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Peter,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks for your reply.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">What I’d really like is to understand how to tune nginx to avoid the delays when I run my tests.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am comfortable with the overly optimistic results from my current “closed model” test design. Once I determine my system’s throughput limits I will introduce significant think times into my scripts so that much larger user populations
are required to produce the same work demand. This will more closely approximate an “open model” test design.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Could you provide more explanation as to why a different load generation tool would avoid triggering a DDOS response from nginx? My first guess would have been that they would also generate requests from a single IP address, and thus look
the same as a JMeter load.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I did try my test with JMeter driving workload from 2 different machines at the same time. I ran each machine ‘s workload at a low enough level that individually they did not trigger the 1 second delay. The combined workload did trigger
the delay for each of the JMeter workload generators. I’m not sure how many machines would be required to avoid the collective response from nginx.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">John<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> nginx [mailto:nginx-bounces@nginx.org] <b>On Behalf Of
</b>Peter Booth<br>
<b>Sent:</b> Monday, March 26, 2018 3:57 PM<br>
<b>To:</b> nginx@nginx.org<br>
<b>Subject:</b> Re: Nginx throttling issue?<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">You’re correct that this is the ddos throttling. The real question is what do you want to do? JMeter with zero think time is an imperfect load generator- this is only one complication. The bigger one is the open/closed model issue. With
you design you have back ptesssure from your system under test to your load generator. A jmeter virtual user will only ever issue a request when the prior one completes. Real users are not so well behaved which is why your test results will always be over
optimistic with this design.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Better approach us to use a load generator that replicates the desired request distribution without triggering the ddos protection. Wrk2, Tsung, httperf are candidates, as well as the cloud based load generator services. Also see Neil Gunther’s
paper on how to combine multiple jmeter instances to replicate real world tragic patterns.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Peter<o:p></o:p></p>
<div id="AppleMailSignature">
<p class="MsoNormal">Sent from my iPhone<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
On Mar 26, 2018, at 4:21 PM, John Melom <<a href="mailto:John.Melom@spok.com">John.Melom@spok.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I am load testing our system using Jmeter as a load generator. We execute a script consisting of an https request executing in a loop. The loop does not contain a think time, since at this point I am not trying to emulate a “real user”.
I want to get a quick look at our system capacity. Load on our system is increased by increasing the number of Jmeter threads executing our script. Each Jmeter thread references different data.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Our system is in AWS with an ELB fronting Nginx, which serves as a reverse proxy for our Docker Swarm application cluster.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">At moderate loads, a subset of our https requests start experiencing to a 1 second delay in addition to their normal response time. The delay is not due to resource contention. System utilizations remain low. The response times cluster
around 4 values: 0 millilseconds, 50 milliseconds, 1 second, and 1.050 seconds. Right now, I am most interested in understanding and eliminating the 1 second delay that gives the clusters at 1 second and 1.050 seconds.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">The attachment shows a response time scatterplot from one of our runs. The x-axis is the number of seconds into the run, the y-axis is the response time in milliseconds. The plotted data shows the response time of requests at the time
they occurred in the run.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">If I run the test bypassing the ELB and Nginx, this delay does not occur.
<o:p></o:p></p>
<p class="MsoNormal">If I bypass the ELB, but include Nginx in the request path, the delay returns.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">This leads me to believe the 1 second delay is coming from Nginx.
<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">One possible candidate Nginx DDOS. Since all requests are coming from the same Jmeter system, I expect they share the same originating IP address. I attempted to control DDOS throttling by setting limit_req as shown in the nginx.conf
fragment below:<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">http {<o:p></o:p></p>
<p class="MsoNormal">…<o:p></o:p></p>
<p class="MsoNormal"> limit_req_zone $binary_remote_addr zone=perf:20m rate=10000r/s;<o:p></o:p></p>
<p class="MsoNormal">…<o:p></o:p></p>
<p class="MsoNormal"><span style="color:black"> server {</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black">…</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black"> location /myReq {</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black"> limit_req zone=perf burst=600;</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black"> proxy_pass xxx.xxx.xxx.xxx;</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black"> }</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black">….</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black"> }</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black">The thinking behind the values set in this conf file is that my aggregate demand would not exceed 10000 requests per second, so throttling of requests should not occur. If there were short bursts more intense
than that, the burst value would buffer these requests.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black">This tuning did not change my results. I still get the 1 second delay.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black">Am I implementing this correctly?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black">Is there something else I should be trying?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black">The responses are not large, so I don’t believe limit_req is the answer.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black">I have a small number of intense users, so limit_conn does not seem likely to be the answer either.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black">Thanks,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black">John Melom</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black">Performance Test Engineer</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:red">Spōk, Inc.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#7F7F7F">+1 (952) 230 5311 <i>Office</i></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#7F7F7F"><a href="mailto:John.Melom@spok.com">John.Melom@spok.com</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#7F7F7F"> </span><o:p></o:p></p>
<p class="MsoNormal"><a href="http://info.spok.com/spokmobilevid"><span style="color:#1F497D;text-decoration:none"><image003.jpg></span></a><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" align="center">
</div>
<p class="MsoNormal"><b><i>NOTE: This email message and any attachments are for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If
you have received this e-mail in error, please contact the sender by replying to this email, and destroy all copies of the original message and any material included with this email.</i></b>
<o:p></o:p></p>
</div>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"><rawRespScatterplot.png><o:p></o:p></p>
</div>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx">http://mailman.nginx.org/mailman/listinfo/nginx</a><o:p></o:p></p>
</div>
</blockquote>
</div>
</div>
<br>
<hr>
<i><b>NOTE: This email message and any attachments are for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you have received this
e-mail in error, please contact the sender by replying to this email, and destroy all copies of the original message and any material included with this email.</b><i></i></i>
</body>
</html>