limit_req for spiders only

Sylvia nginx-forum at nginx.us
Mon Oct 14 13:25:24 UTC 2013


Hello.

Doesnt robots.txt "Crawl-Delay" directive satisfy your needs? 
Normal spiders should obey robots.txt, if they dont - they can be banned.

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,243670,243674#msg-243674



More information about the nginx mailing list