<html><head></head><body lang="en-US" style="background-color: rgb(255, 255, 255); line-height: initial;"> <div style="width: 100%; font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif, sans-serif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);">That looks promising. </div><div style="width: 100%; font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif, sans-serif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);"><span style="font-family: Calibri, 'Slate Pro', sans-serif, sans-serif; font-size: initial; text-align: initial; line-height: initial;"><br></span></div><div style="width: 100%; font-size: initial; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);"><span style="font-size: initial; text-align: initial; line-height: initial;"><font face="Calibri, Slate Pro, sans-serif, sans-serif">BTW most Google Image bots are fake. But I don't allow hot linking. A legitimate Google user viewing the reduced resolution image provided by Google can click to see the referring page, so no loss to real users. </font></span></div><div style="width: 100%; font-size: initial; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);"><span style="font-size: initial; text-align: initial; line-height: initial;"><font face="Calibri, Slate Pro, sans-serif, sans-serif"><br></font></span></div><div style="width: 100%; font-size: initial; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);"><span style="font-size: initial; text-align: initial; line-height: initial;"><font face="Calibri, Slate Pro, sans-serif, sans-serif">That hacker was quite insistent. I got a 414 (large request) for the first time. Perhaps a buffer overflow attempt. </font></span></div> <div style="width: 100%; font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif, sans-serif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);"><br style="display:initial"></div> <div style="font-size: initial; font-family: Calibri, 'Slate Pro', sans-serif, sans-serif; color: rgb(31, 73, 125); text-align: initial; background-color: rgb(255, 255, 255);"></div> <table width="100%" style="background-color:white;border-spacing:0px;"> <tbody><tr><td colspan="2" style="font-size: initial; text-align: initial; background-color: rgb(255, 255, 255);"> <div style="border-style: solid none none; border-top-color: rgb(181, 196, 223); border-top-width: 1pt; padding: 3pt 0in 0in; font-family: Tahoma, 'BB Alpha Sans', 'Slate Pro'; font-size: 10pt;"> <div><b>From: </b>Wandenberg Peixoto</div><div><b>Sent: </b>Sunday, September 25, 2016 3:05 PM</div><div><b>To: </b>nginx@nginx.org</div><div><b>Reply To: </b>nginx@nginx.org</div><div><b>Subject: </b>Re: fake googlebots</div></div></td></tr></tbody></table><div style="border-style: solid none none; border-top-color: rgb(186, 188, 209); border-top-width: 1pt; font-size: initial; text-align: initial; background-color: rgb(255, 255, 255);"></div><br><div id="_originalContent" style=""><div dir="ltr">Some time ago I wrote this <a href="https://github.com/wandenberg/nginx-trusted-proxy-resolver-module">module</a> to check when an access is done through the Google Proxy using reverse DNS + DNS resolve and comparing the results to validate the access.<div>You can do something similar.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Sep 25, 2016 at 11:58 PM, <a href="mailto:lists@lazygranch.com">lists@lazygranch.com</a> <span dir="ltr"><<a href="mailto:lists@lazygranch.com" target="_blank">lists@lazygranch.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I got a spoofed googlebot hit. It was easy to detect since there were<br>
probably a hundred requests that triggered my hacker detection map<br>
scheme. Only two requests received a 200 return and both were harmless.<br>
<br>
200 118.193.176.53 - - [25/Sep/2016:17:45:23 +0000] "GET / HTTP/1.1" 847 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +<a href="http://www.google.com/bot.html" rel="noreferrer" target="_blank">http://www.google.com/bot.<wbr>html</a>)" "-"<br>
<br>
For the fake googlebot:<br>
# host 118.193.176.53<br>
Host 53.176.193.118.in-addr.arpa not found: 3(NXDOMAIN)<br>
<br>
For a real googlebot:<br>
# host 66.249.69.184<br>
184.69.249.66.in-addr.arpa domain name pointer <a href="http://crawl-66-249-69-184.googlebot.com" rel="noreferrer" target="_blank">crawl-66-249-69-184.googlebot.<wbr>com</a>.<br>
<br>
IP2location shows it is a Chinese ISP:<br>
3(NXDOMAIN)<a href="http://www.ip2location.com/118.193.176.53" rel="noreferrer" target="_blank">http://www.<wbr>ip2location.com/118.193.176.53</a><br>
<br>
Nginx has a reverse DNS module:<br>
<a href="https://github.com/flant/nginx-http-rdns" rel="noreferrer" target="_blank">https://github.com/flant/<wbr>nginx-http-rdns</a><br>
I see it has a 10.1 issue:<br>
<a href="https://github.com/flant/nginx-http-rdns/issues/8" rel="noreferrer" target="_blank">https://github.com/flant/<wbr>nginx-http-rdns/issues/8</a><br>
<br>
Presuming this bug gets fixed, does anyone have code to verify<br>
googlebots? Or some other method?<br>
<br>
______________________________<wbr>_________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx" rel="noreferrer" target="_blank">http://mailman.nginx.org/<wbr>mailman/listinfo/nginx</a><br>
</blockquote></div><br></div>
<br><!--end of _originalContent --></div></body></html>