rate limit with good bot IPs whitelisted

[neubyr]

I am trying to figure out if there is any way to rate limit all traffic
except Googlebot, msnbot, yandex and baidu bots. Here is what I have
started with:


  # Whitelisted IPs
  geo $rate_limit_ip {
      default $binary_remote_addr;
      127.0.0.1 "";
      10.0.0.0/8 "";
  }

  # Rate limit
  limit_req_zone $rate_limit_ip zone=publix:10m rate=10r/s;

I can add googlebot, msnbot, yandex and baidu IP ranges manually to the
whitelist, but that will make lookup table big. I am not sure whether this
approach will work for high traffic like - 1200 requests/second distributed
across 20 nginx hosts. Any ideas on such setup will be really helpful.

Also, can such host lookups be done in real-time for every request? I am
guessing that may not be efficient for each request, but I was wondering if
there are any solutions.

Appreciate all your help.

- thanks,
N
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20141121/7f11bd5e/attachment.html>