block robots for some urls

[bvidinli]

1. In robotstxt.org site, it says:
"robots can ignore your /robots.txt. Especially malware robots that
scan the web for security vulnerabilities, and email address
harvesters used by spammers will pay no attention."

2. I am expeirencing nginx and learning its rewrite features.
So, I try to write rewrite statements.

still i have no luck.


2010/7/19 Nuno Magalhães <nunomagalhaes at eu.ipp.pt>:
> On Sun, Jul 18, 2010 at 22:34, toto2008 <nginx-forum at nginx.us> wrote:
>> Hello,
>>
>> Why not using a robots.txt file ? http://www.robotstxt.org/
>
> It won't block bots that don't honor it (not would anything block bots
> that present themselves as mozilla).
>
> --
> ()  ascii-rubanda kampajno - kontraŭ html-a retpoŝto
> /\  ascii ribbon campaign - against html e-mail
>
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://nginx.org/mailman/listinfo/nginx
>