block robots for some urls
bvidinli at gmail.com
Mon Jul 19 03:40:54 MSD 2010
1. In robotstxt.org site, it says:
"robots can ignore your /robots.txt. Especially malware robots that
scan the web for security vulnerabilities, and email address
harvesters used by spammers will pay no attention."
2. I am expeirencing nginx and learning its rewrite features.
So, I try to write rewrite statements.
still i have no luck.
2010/7/19 Nuno Magalhães <nunomagalhaes at eu.ipp.pt>:
> On Sun, Jul 18, 2010 at 22:34, toto2008 <nginx-forum at nginx.us> wrote:
>> Why not using a robots.txt file ? http://www.robotstxt.org/
> It won't block bots that don't honor it (not would anything block bots
> that present themselves as mozilla).
> () ascii-rubanda kampajno - kontraŭ html-a retpoŝto
> /\ ascii ribbon campaign - against html e-mail
> nginx mailing list
> nginx at nginx.org
More information about the nginx