0.7.67 - limit_req problems

[zuzur]

Hello all,

There must be something that i miss when trying to setup rate limiting
on our 0.7.67 installation

We are using nginx as a load-balancer for a rails application, and
recently we have been hit by a very badly written bot that generates up
to 10 req/s without checking the results. Much like a e-mail harvesting
bot, very badly written ...

The first thing i did was to make sure their requests weren't forwarded
to the backend, issuing a 444 and hoping that they would notice and just
stop doing what they are unsuccessfully trying to do :-) 

That didn't stop them. So currently, i am blacklisting their IPs each
time i see that bot reappearing. Guess what ... they are on EC2
instances and seem to be very flexible when it comes to changing their
IP address :-) 

I grew a bit tired after twelve attempts, so i'm looking for alternate
possibilities. Note that we have some legitimate traffic on the same
server that can reach that amount of requests, so i can't just blindly
apply global rate limiting or add rules to the firewall.

Their IP can change, they are using a very specific user-agent, and are
hitting a very specific location that isn't normally available on this
server, so i came up with the following config:

[code]
http {
  include        /opt/nginx/conf/mime.types;
  default_type   application/octet-stream;

  geoip_country  /opt/nginx/conf/GeoIP.dat;
  geoip_city     /opt/nginx/conf/GeoLiteCity.dat;

  limit_req_zone $binary_remote_addr zone=one:10m rate=2r/s;
  limit_req_zone $http_user_agent zone=blackhole:10m rate=1r/m;
  limit_zone     two $binary_remote_addr 10m;
  ...
  server {
  ...
    location ^~ /api/ {
      limit_req zone=blackhole;
      limit_conn two 1;
      return 444;
    }
 ...
[/code]


But it doesn't seem to work, my server is still hit with many req/s,
much more than what i specified in the configuration. 

What am i doing wrong here ? do limit_* directives really work in
"location" directives ?

What i ultimately would like to do is implement some kind of blackhole
for those guys:
- make their request sit there idling for a long time
- make sure they don't take too much simultaneous connections, so we
have workers available for legit requests (limit_conn

Is there such a module available ?

Thanks a lot in advance,

Erwan

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,98958,98958#msg-98958