block robots for some urls

Nuno Magalhães nunomagalhaes at eu.ipp.pt
Mon Jul 19 01:41:30 MSD 2010


On Sun, Jul 18, 2010 at 22:34, toto2008 <nginx-forum at nginx.us> wrote:
> Hello,
>
> Why not using a robots.txt file ? http://www.robotstxt.org/

It won't block bots that don't honor it (not would anything block bots
that present themselves as mozilla).

-- 
()  ascii-rubanda kampajno - kontraŭ html-a retpoŝto
/\  ascii ribbon campaign - against html e-mail



More information about the nginx mailing list