<div dir="ltr">This is almost certainly not Google as they obey robots.txt. The & to & conversion is another sign of a poor quality crawler. Check the RDNS and you will find it's probably some IP faking Google UA, I suggest blocking at network level.</div><br><div class="gmail_quote"><div dir="ltr">On Fri, Jun 8, 2018 at 1:57 AM shiz <<a href="mailto:nginx-forum@forum.nginx.org">nginx-forum@forum.nginx.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
Recently, Google has started spidering my website and in addition to normal<br>
pages, appended "&" to all urls, even the pages excluded by robots.txt<br>
<br>
e.g. page.php?page=aaa -> page.php?page=aaa&<br>
<br>
Any idea how to redirect/rewrite this?<br>
<br>
Posted at Nginx Forum: <a href="https://forum.nginx.org/read.php?2,280093,280093#msg-280093" rel="noreferrer" target="_blank">https://forum.nginx.org/read.php?2,280093,280093#msg-280093</a><br>
<br>
_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx" rel="noreferrer" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a><br>
</blockquote></div>