Calculating requests per second, per IP address

[Maxim Dounin]

Hello!

On Fri, Dec 29, 2023 at 09:54:30AM -0300, Rejaine Monteiro wrote:

> Hi all,
> 
> I´m running Nginx community edition and need to implement rate limiting
> 
> There's plenty of guides out there on how to do this, but no guides on how
> to get real values/stats from the access logs
> 
> 
> What I need to get from the NGINX access logs is:
> 
> - Requests per minute, per IP address
> 
> - Average requests per minute, derived from all IP addresses
> 
> - Max requests per minute, per IP address
> 
> We have a few endpoints with different functionalities and we simply cannot
> have a common rule that works for everyone.
> 
> Any tips on a tool or script that could help generate this type of
> information (if possible in real time or collect this data for future
> analysis)?

Note that nginx's limit_req module implements limiting based on 
the "leaky bucket" algorithm, which permits traffic bursts when 
properly configured, and therefore there is no need to calculate 
exact rates of requests per minute and so on.

Rather, I would recommend to focus on the number of requests a 
typical legitimate user can generate to the protected resources 
within a short period of time, and set large enough burst= value 
to ensure legitimate users are not limited.  Then use some 
reasonable long-term rate - this does not need to be exact, but 
rather somewhat reasonable, for example, to cover typical number 
of requests per day from a legitimate user.

Note well that using "nodelay" (or "delay=N") is recommended with 
such approach, see http://nginx.org/r/limit_req for details.

-- 
Maxim Dounin
http://mdounin.ru/