0.7.67 - limit_req problems

[Maxim Dounin]

Hello!

On Wed, Jun 16, 2010 at 09:46:21AM -0400, zuzur wrote:

> Hello all,
> 
> There must be something that i miss when trying to setup rate limiting
> on our 0.7.67 installation
> 
> We are using nginx as a load-balancer for a rails application, and
> recently we have been hit by a very badly written bot that generates up
> to 10 req/s without checking the results. Much like a e-mail harvesting
> bot, very badly written ...
> 
> The first thing i did was to make sure their requests weren't forwarded
> to the backend, issuing a 444 and hoping that they would notice and just
> stop doing what they are unsuccessfully trying to do :-) 
> 
> That didn't stop them. So currently, i am blacklisting their IPs each
> time i see that bot reappearing. Guess what ... they are on EC2
> instances and seem to be very flexible when it comes to changing their
> IP address :-) 
> 
> I grew a bit tired after twelve attempts, so i'm looking for alternate
> possibilities. Note that we have some legitimate traffic on the same
> server that can reach that amount of requests, so i can't just blindly
> apply global rate limiting or add rules to the firewall.
> 
> Their IP can change, they are using a very specific user-agent, and are
> hitting a very specific location that isn't normally available on this
> server, so i came up with the following config:
> 
> [code]
> http {
>   include        /opt/nginx/conf/mime.types;
>   default_type   application/octet-stream;
> 
>   geoip_country  /opt/nginx/conf/GeoIP.dat;
>   geoip_city     /opt/nginx/conf/GeoLiteCity.dat;
> 
>   limit_req_zone $binary_remote_addr zone=one:10m rate=2r/s;
>   limit_req_zone $http_user_agent zone=blackhole:10m rate=1r/m;
>   limit_zone     two $binary_remote_addr 10m;
>   ...
>   server {
>   ...
>     location ^~ /api/ {
>       limit_req zone=blackhole;
>       limit_conn two 1;
>       return 444;
>     }
>  ...
> [/code]
> 
> 
> But it doesn't seem to work, my server is still hit with many req/s,
> much more than what i specified in the configuration. 

In /api/ location you explicitly return 444 for all users.  It's 
not clear what effect do you expect from limit_req/limit_conn 
there but most likely it will be negative - requests will be 
delayed / or 503 will be returned instead of immediate connection 
close.

> What am i doing wrong here ? do limit_* directives really work in
> "location" directives ?

Yes they do.

Maxim Dounin