limit-req and greedy UAs

[lists at lazygranch.com]

‎https://www.nginx.com/blog/tuning-nginx/

‎I have far more faith in this write up regarding tuning than the anti-ddos, though both have similarities. 

My interpretation is the user bandwidth is connections times rate. But you can't limit the connection to one because (again my interpretation) there can be multiple users behind one IP. Think of a university reading your website. Thus I am more comfortable limiting bandwidth than I am limiting the number of connections. ‎The 512k rate limit is fine. I wouldn't go any higher. 

I don't believe their is one answer here because it depends on how the user interacts with the website. I only serve static content. In fact, I only allow the verbs "head" and "get" to limit the attack surface. A page of text and photos itself can be many things. Think of a photo gallery versus a forum page. The forum has mostly text sprinkled with avatar photos, while the gallery can be mostly images with just a line of text each. 

Basically you need to experiment. Even then, your setup may be better or worse than the typical user. That said, if you limited the rate to 512k bytes per second, most users could achieve that rate‎. 

I just don't see evidence of download managers. I see plenty of wget, curl, and python. Those people get my 444 treatment. I use the map module as indicated in my other post to do this. 

What I haven't mentioned is filtering out machines. If you are really concerned about your system being overloaded, think about the search engines you want to support. Assuming you want Google, you need to set up your website in a manner so that Google knows you own it, then you can throttle it back. Google is maybe 20% of my referrals.

If you have a lot of photos, you can set up nginx to block hit linking. This is significant because Google images will hot link everything you have. What you want is for Google itself to see your images, which it will present in reduced resolution, but block the Google hot link. If someone really wants to see your image, Google supplies the referal page. 

http://nginx.org/en/docs/http/ngx_http_referer_module.html

I make my own domain a valid, but maybe that is assumed. If you want to place a link to an image on your website in a forum, you need to make that forum valid. 

Facebook will steal your images.
http://badbots.vps.tips/info/facebookexternalhit-bot

I would use the nginx map module since you will probably be blocking many bots. 

Finally, you may want to block "the cloud"‎ using your firewall. Only block the browser ports since mail servers will be on the cloud. I block all of AWS for example. My nginx.conf also flags certain requests such as logging into WordPress since I'm not using WordPress! Clearly that IP is a hacker. I have plenty more signatures in the map. I have a script that pulls the IP addresses out of the access.log. I get maybe 20 addresses a day. I feed them to ip2location. Any address that goes to a cloud, VPS, colo, hosting company gets added to the firewall blocking list. I don't just block the IP, but I use the Hurricane Electric BGP tool to get the entire IP space to block. As a rule, I don't block schools, libraries, or ISPs. The idea here is to allow eyeballs but not machines. 

You can also use commercial blocking services if you trust them. (I don't. )


  Original Message  
From: Grant
Sent: Sunday, September 11, 2016 10:28 AM
To: nginx at nginx.org
Reply To: nginx at nginx.org
Subject: Re: limit-req and greedy UAs

> ‎This page has all the secret sauce, including how to limit the number of connections.
>
> https://www.nginx.com/blog/mitigating-ddos-attacks-with-nginx-and-nginx-plus/
>
> I set up the firewall with a higher number as a "just in case."


Should I basically duplicate my limit_req and limit_req_zone
directives into limit_conn and limit_conn_zone? In what sort of
situation would someone not do that?

- Grant

_______________________________________________
nginx mailing list
nginx at nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx