<html><head></head><body lang="en-US" style="background-color: rgb(255, 255, 255); line-height: initial;"> <div style="width: 100%; font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif, sans-serif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);">But again, if you have a particular IP behaving badly, and limiting for that IP kicks in, I say they get what they deserve. The goal of limiting resources to one particular IP is to prevent other users from having degraded service. </div><div style="width: 100%; font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif, sans-serif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);"><br></div><div style="width: 100%; font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif, sans-serif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);">That said, I did set up my limiting for each location with the idea that I could at some point fine tune things. </div><div style="width: 100%; font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif, sans-serif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);"><br></div><div style="width: 100%; font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif, sans-serif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);">Trawling the interwebs, request limiting shows up in anti-ddos "hints" for nginx.conf set up. In reality, I doubt they do much good for a real distributed attack, but they do limit a particular IP from hogging resources. (One IP could be many different eyeballs sitting behind a router, so you don't want to get too restrictive.)</div><div style="width: 100%; font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif, sans-serif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);"><br></div><div style="width: 100%; font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif, sans-serif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);">What looks to me to be a real resource hog that quite frankly you cant do much about are download managers. They open up multiple connections, but the rate limits apply to each individual connection. (this is why you want to limit the number of connections.) You can detect their user agents. I set up my nginx.conf to return a 444 when I see a wget, python, curl, etc. I haven't bothered to flag go!zilla or all the known download managers because I don't see evidence of them being used. </div><div style="width: 100%; font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif, sans-serif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);"><br></div><div style="width: 100%; font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif, sans-serif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);">I have scripts to pull the "444" hits off the access log. Any IP that is not from "eyeballs" such as VPS, colo, etc. gets added to my firewall blocking list. </div> <div style="width: 100%; font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif, sans-serif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);"><br style="display:initial"></div> <div style="font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif, sans-serif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);"></div> <table width="100%" style="background-color:white;border-spacing:0px;"> <tbody><tr><td colspan="2" style="font-size: initial; text-align: initial; background-color: rgb(255, 255, 255);"> <div style="border-style: solid none none; border-top-color: rgb(181, 196, 223); border-top-width: 1pt; padding: 3pt 0in 0in; font-family: Tahoma, 'BB Alpha Sans', 'Slate Pro'; font-size: 10pt;"> <div><b>From: </b>Richard Stanway</div><div><b>Sent: </b>Friday, September 9, 2016 6:01 AM</div><div><b>To: </b>nginx@nginx.org</div><div><b>Reply To: </b>nginx@nginx.org</div><div><b>Subject: </b>Re: limit-req and greedy UAs</div></div></td></tr></tbody></table><div style="border-style: solid none none; border-top-color: rgb(186, 188, 209); border-top-width: 1pt; font-size: initial; text-align: initial; background-color: rgb(255, 255, 255);"></div><br><div id="_originalContent" style=""><div dir="ltr">You can put limit_req in a location, for example do not limit static files and only limit expensive backend hits, or use two different thresholds.</div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Sep 9, 2016 at 3:39 AM, <span dir="ltr"><<a href="mailto:lists@lazygranch.com" target="_blank">lists@lazygranch.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Since this limit is per IP, is the scenario you stated really a problem? Only that IP is effected. Or as is often the case, did I miss something?<br>
<br>
<a href="http://nginx.org/en/docs/http/ngx_http_limit_req_module.html" rel="noreferrer" target="_blank">http://nginx.org/en/docs/http/<wbr>ngx_http_limit_req_module.html</a><br>
<br>
Original Message <br>
From: Grant<br>
Sent: Thursday, September 8, 2016 6:24 PM<br>
To: <a href="mailto:nginx@nginx.org">nginx@nginx.org</a><br>
Reply To: <a href="mailto:nginx@nginx.org">nginx@nginx.org</a><br>
Subject: limit-req and greedy UAs<br>
<div class="HOEnZb"><div class="h5"><br>
Has anyone considered the problem of legitimate UAs which request a<br>
series of files which don't necessarily exist when they access your<br>
site? Requests for files like robots.txt, sitemap.xml,<br>
crossdomain.xml, apple-touch-icon.png, etc could quickly cause the UA<br>
to exceed the limit-req burst value. What is the right way to deal<br>
with this?<br>
<br>
- Grant<br>
<br>
______________________________<wbr>_________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx" rel="noreferrer" target="_blank">http://mailman.nginx.org/<wbr>mailman/listinfo/nginx</a><br>
<br>
______________________________<wbr>_________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx" rel="noreferrer" target="_blank">http://mailman.nginx.org/<wbr>mailman/listinfo/nginx</a></div></div></blockquote></div><br></div>
<br><!--end of _originalContent --></div></body></html>