fake googlebots

[Rainer Duffner]

> Am 25.09.2016 um 23:58 schrieb lists at lazygranch.com:
> 
> I got a spoofed googlebot hit. It was easy to detect since there were
> probably a hundred requests that triggered my hacker detection map
> scheme. Only two requests received a 200 return and both were harmless.
> 
> 200 118.193.176.53 - - [25/Sep/2016:17:45:23 +0000] "GET / HTTP/1.1" 847 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"
> 
> For the fake googlebot:
> # host 118.193.176.53
> Host 53.176.193.118.in-addr.arpa not found: 3(NXDOMAIN)
> 
> For a real googlebot:
> # host 66.249.69.184
> 184.69.249.66.in-addr.arpa domain name pointer crawl-66-249-69-184.googlebot.com.
> 
> IP2location shows it is a Chinese ISP:
> 3(NXDOMAIN)http://www.ip2location.com/118.193.176.53
> 
> Nginx has a reverse DNS module:
> https://github.com/flant/nginx-http-rdns
> I see it has a 10.1 issue:
> https://github.com/flant/nginx-http-rdns/issues/8
> 
> Presuming this bug gets fixed, does anyone have code to verify
> googlebots? Or some other method?




Sorry to be so blunt - but what’s the point?

You can also password-protect your site and give the credentials only to your friends.
Problem solved.

Most of the traffic of the web these days is created by bots (unless you’re a popular shop or offer original, often updated content for popular topics, then you’ll actually get visitors).

If it’s not the Big G, it might be bing or baidu or yandex or some other bot.