Serve *only* from cache for particular user-agents
list_nginx at bluerosetech.com
Fri Feb 21 22:15:06 UTC 2014
On 2/21/2014 7:25 AM, rge3 wrote:
> I havne't found any ideas for this and thought I might ask here. We have a
> fairly straightforward proxy_cache setup with a proxy_pass backend. We
> cache documents for different lengths of time or go the backend for what's
> missing. My problem is we're getting overrun with bot and spider requests.
> MSN in particular started hitting us exceptionally hard yesterday and
> started bringing our backend servers down. Because they're crawling the
> site from end to end our cache is missing a lot of those pages and nginx has
> to pass the request on through.
Are they ignoring your robots.txt?
More information about the nginx