<html><head><style id="outgoing-font-settings">#response_container_BBPPID{font-family: initial;font-size: initial;color: initial;}</style></head><body style="background-color: rgb(255, 255, 255); background-image: initial; line-height: initial;"><div id="response_container_BBPPID" style="outline:none" dir="auto" contenteditable="false"> <div id="BB10_response_div_BBPPID" dir="auto" style="width:100%">There are probably 50 common web crawlers. If they aren't Google, Apple, or Microsoft, I don't want them. The worst is one called "majestic 12". It seems to suck down the the entire website every visit. There are some that try to determine what ads your serve, of which I serve none. Another reads your website to collect text to see if students are copying your website for term papers. It goes on an on and quite frankly if they aren't a major search engine they are just slowing down your actual viewers. The Chinese search engines don't give a crap about robots.txt. Bots are maybe 40% of internet traffic. </div><div id="BB10_response_div_BBPPID" dir="auto" style="width:100%"><br></div><div id="BB10_response_div_BBPPID" dir="auto" style="width:100%">Note these reports try to make the internet look worse than it is because they are selling a service. </div><div id="BB10_response_div_BBPPID" dir="auto" style="width:100%">https://www.imperva.com/blog/bad-bot-report-2021-the-pandemic-of-the-internet/</div><div id="BB10_response_div_BBPPID" dir="auto" style="width:100%"><br></div><div id="BB10_response_div_BBPPID" dir="auto" style="width:100%">I have plenty of transit capacity. I can serve 3TB a month and I do 30GB. What I don't have is CPU power. I have a one CPU VPS. The CPU is shared resource. I think the RAM used by the VPS is more "available," if that makes any sense. That is they don't swap you out but let your VPS sit in RAM. So something RAM intensive is fine but CPU intensive is not. </div><div id="BB10_response_div_BBPPID" dir="auto" style="width:100%"><br></div><div id="BB10_response_div_BBPPID" dir="auto" style="width:100%">I run nginx plus my personal email via postfix and Dovecot. The website serves static pages. I use no CMS. I write with a WYSIWG editor. So bare nginx is very CPU friendly. </div><div id="BB10_response_div_BBPPID" dir="auto" style="width:100%"><br></div><div id="BB10_response_div_BBPPID" dir="auto" style="width:100%">The lack of CMS make it really hard to hack. I use no cpanel or any web interface to control the website. The bots will password guess if they find a service that uses passwords. Everything I do to control the website is by command line via ssh. Maintaining a low attack surface makes it hard to hack. </div><div id="BB10_response_div_BBPPID" dir="auto" style="width:100%"><br></div><div id="BB10_response_div_BBPPID" dir="auto" style="width:100%">Ssh is under constant attack. Use a PKI rather than password. </div><div id="BB10_response_div_BBPPID" dir="auto" style="width:100%"><br></div><div id="BB10_response_div_BBPPID" dir="auto" style="width:100%">You should check the nginx access log for return code 400. Nginx can detect and obviously block some hacking. I never found information on what triggers a 400 but I assure you there are no false positives. I read all the 400 returns because I don't control the trigger so I know the lack of false positives as a fact. There aren't many but they are all hacker related. </div><div id="BB10_response_div_BBPPID" dir="auto" style="width:100%"><br></div><div id="BB10_response_div_BBPPID" dir="auto" style="width:100%">It you set up the nginx access log format with the return code as the first parameter, you can easily write scripts to pull lines by return code by using the "^" in the pattern match to match the number at the start of a line. </div><div id="BB10_response_div_BBPPID" dir="auto" style="width:100%"><br></div><div id="BB10_response_div_BBPPID" dir="auto" style="width:100%">You should register your website with Google. You need to place a file on your website that they provide. You can force Google to do a crawl if you added significant content. </div><div id="BB10_response_div_BBPPID" dir="auto" style="width:100%"><br></div><div id="BB10_response_div_BBPPID" dir="auto" style="width:100%">Nginx can block hot linking. That shows up as 403. You will find Twitter and Facebook users will find some photo and pass it around. Unless you caption every photo this will not drive eyeballs to your website. I use nginx to block hot linking photos and pdfs. </div><div id="BB10_response_div_BBPPID" dir="auto" style="width:100%"><br></div><div id="BB10_response_div_BBPPID" dir="auto" style="width:100%">If you block server IP space you will be blocking VPNs. Now what you might want to do is if you don't block the IP space is then give them a captcha. Much hacking comes from VPNs other than Opera which I don't block. The new Apple privacy feature has dedicated 3rd party IP space that you don't want to block. They document it. </div> <div id="response_div_spacer_BBPPID" dir="auto" style="width:100%"> <br></div><div id="response_div_spacer_BBPPID" dir="auto" style="width:100%">It is nasty out there. Less is more. Keep it simple. </div> <div id="blackberry_signature_BBPPID" dir="auto"> <div id="_signaturePlaceholder_BBPPID" dir="auto"></div> </div></div><div id="_original_msg_header_BBPPID" dir="auto"> <table id="_pHCWrapper_BBPPID" width="100%" style="border-spacing: 0px; display: table; outline: none;" contenteditable="false"><tbody><tr><td colspan="2"> <div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb( 181 , 196 , 223 );padding:3pt 0in 0in;font-family:'tahoma' , 'bb alpha sans' , 'slate pro';font-size:10pt"> <div id="from"><b>From:</b> skip.montanaro@gmail.com</div><div id="sent"><b>Sent:</b> February 14, 2022 12:30 PM</div><div id="to"><b>To:</b> nginx@nginx.org</div><div id="reply_to"><b>Reply-to:</b> nginx@nginx.org</div><div id="subject"><b>Subject:</b> Re: Obvious malware rejection module?</div></div></td></tr></tbody></table> <br> </div><div dir="auto" style="outline:none" contenteditable="false"><div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:'arial' , sans-serif">Thanks for the reply. I actually am interested in bots, at least in the form of legitimate web crawlers. My itty bitty website was specifically intended to expose the archives of a defunct (but formerly public) mailing list to interested parties and said crawlers:</div><div class="gmail_default" style="font-family:'arial' , sans-serif"><br></div><div class="gmail_default" style="font-family:'arial' , sans-serif"><a href="https://smontanaro.net/CR/">https://smontanaro.net/CR/</a><br></div><div class="gmail_default" style="font-family:'arial' , sans-serif"><br></div><div class="gmail_default" style="font-family:'arial' , sans-serif">Performance requirements are decidedly minimal.</div><div class="gmail_default" style="font-family:'arial' , sans-serif"><br></div><div class="gmail_default" style="font-family:'arial' , sans-serif">In addition, the last time I did anything with the web was about 20 years ago, so this whole enterprise is a fun (for some definition of "fun") reacquaintance with more current web application development. I spent most of my career (I'm now retired) using Python to implement various things for various uses, so Flask was a pretty straightforward way to get something up and running quickly.</div><div class="gmail_default" style="font-family:'arial' , sans-serif"><br></div><div class="gmail_default" style="font-family:'arial' , sans-serif">As a placeholder for something more sophisticated, a couple location directives squashed a large fraction of the problematic URIs.</div><div class="gmail_default" style="font-family:'arial' , sans-serif"><br></div><div class="gmail_default" style="font-family:'arial' , sans-serif">Skip Montanaro</div></div></div>
</div></body></html>