Recommended limit_req and limit_conn for location ~ \.php$ {}

[c0nw0nk]

Francis Daly Wrote:
-------------------------------------------------------
> On Mon, Sep 26, 2016 at 07:41:12PM -0400, c0nw0nk wrote:
> 
> Hi there,
> 
> > Whats a good setting that won't effect legitimate decent (I think I
> just
> > committed a crime calling some of these companies decent?) crawlers
> like
> > Google, Bing, Baidu, Yandex etc.
> 
> Look at your logs for traffic from Google, Bing, Baidu, Yandex etc.
> How
> many of these requests do they make per second, when they make
> requests?
> 
> Set your limits to allow those.
> 
> Note that any of the clients can change their crawl-rate on your site
> at
> any time, so any number that worked for you yesterday will not
> necessarily
> work tomorrow.
> 
> Cheers,
> 
> 	f
> -- 
> Francis Daly        francis at daoine.org
> 
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
oscaretu . Wrote:
-------------------------------------------------------
> Perhaps this can help:
> 
> http://stackoverflow.com/questions/12022429/http-status-code-for-overl
> oaded-server
> 
> Another option is a *429 - Too Many Requests* response.
> 
> Defined in RFC6585 - http://tools.ietf.org/html/rfc6585#section-4
> 
> The spec does not define how the origin server identifies the user,
> nor how
> it counts requests.
> 
> For example, an origin server that is limiting request rates can do so
> based upon counts of requests on a per-resource basis, across the
> entire
> server, or even among a set of servers.
> 
> Likewise, it might identify the user by its authentication
> credentials, or
> a stateful cookie.
> 
> Also see the Retry-After header in the response.
> 
> Kind regards,
> Oscar
> 
> 
> On Tue, Sep 27, 2016 at 10:59 AM, itpp2012
> <nginx-forum at forum.nginx.org>
> wrote:
> 
> > General rule of thumb is set it as low as possible, as soon as 503's
> are
> > getting your users upset or resources are getting blocked, then
> double the
> > values, keep an eye on the logs, double it one more time when
> required,
> > done.
> >
> > Posted at Nginx Forum: https://forum.nginx.org/read.
> > php?2,269862,269866#msg-269866
> >
> > _______________________________________________
> > nginx mailing list
> > nginx at nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx
> >
> 
> 
> 
> -- 
> Oscar Fernandez Sierra
> oscaretu at gmail.com
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx


Thanks raises a question of why the default status code on limit_conn and
limit_req is a 503 instead of a 429.

limit_conn_status 503;
limit_req_status 503;

But under a DoS attack I always feel those values would be better being
"444" since the server won't respond and cut's the connection rather than
waste bandwidth on a client who is opening and closing connections fast as a
bullet. You tend to get allot of 499's in your access.log when really you
should not even waste your bandwidth output trying to respond to them.

Is it worth me changing the default status codes or keeping them at
default.

Also I am going to start out with itpp2012's advice setting it at the low
value and increasing it based on what companies crawlers require a more
brutal crawl rate Francis Daly's idea is also good though but since I use
"burst=5" instead of "nodelay" on my request limit I get to find out what
crawlers would actually wait for the current request that they are
experiencing a loading page on to finish.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,269862,269872#msg-269872