Calculating requests per second, per IP address

[J Carter]

Hello,

On Fri, 29 Dec 2023 09:54:30 -0300
Rejaine Monteiro <rejaine at bhz.jamef.com.br> wrote:

> Hi all,
> 
> I´m running Nginx community edition and need to implement rate limiting
> 
> There's plenty of guides out there on how to do this, but no guides on how
> to get real values/stats from the access logs
> 
> 
> What I need to get from the NGINX access logs is:
> 
> - Requests per minute, per IP address
> 
> - Average requests per minute, derived from all IP addresses
> 
> - Max requests per minute, per IP address
> 
> We have a few endpoints with different functionalities and we simply cannot
> have a common rule that works for everyone.
> 
> Any tips on a tool or script that could help generate this type of
> information (if possible in real time or collect this data for future
> analysis)?
> 
> 
> I appreciate any tips.
> 

There isn't an existing bespoke tool for this (at least publicly 
available).

Normally such metrics are generated by:
A) Feeding access logs into a log aggregator platform (Splunk, Loki)
B) Performing / creating custom queries on that platform, to generate
such reports.

Of note, Loki (which is AGPL/free) has a nice user experience for 
this.

https://grafana.com/docs/loki/latest/query/metric_queries/

Other than log aggregators, writing a script (python, perl, bash) is 
likely the fastest approach. Consider sharing it if you do, I'm sure 
others will find it useful.

If you're looking for existing scripts as a starting point, there may 
be  similar tools for Apache that you could adapt for nginx. Both use
the 'Common Log Format' for access logs by default.

https://github.com/ajohnstone/apache-log-stats

Something like this.