Possible limitation of ngx_http_limit_req_module

[Valentin V. Bartenev]

On Monday 11 May 2015 10:59:46 jwroblewski wrote:
> Hi,
> 
> I'm observing an inconsistent behavior of ngx_http_limit_req_module in nginx
> 1.7.12. The relevant excerpts from my config:
> 
> http {
> ...
>     # A fixed string used as a key, to make all requests fall into the same
> zone
>     limit_req_zone test_zone zone=test_zone:1m rate=5r/s;
> ...
>     server {
>     ...
>         location /limit {
>             root /test
>             limit_req zone=test_zone nodelay;
>         }
>     ...
>     }
> }
> 
> I use wrk to hammer the server for 5 secs:
> 
> $ ./wrk -t 100 -c 100 -d 5 http://127.0.0.1/limit/test
> Running 5s test @ http://127.0.0.1/limit/test
>   100 threads and 100 connections
>   Thread Stats   Avg      Stdev     Max   +/- Stdev
>     Latency     2.82ms    2.96ms  15.12ms   88.92%
>     Req/Sec   469.03    190.97     0.89k    62.05%
>   221531 requests in 5.00s, 81.96MB read
>   Non-2xx or 3xx responses: 221506
> Requests/sec:  44344.69
> Transfer/sec:     16.41MB
> 
> So, out of 221531 sent requests, 221506 came back with error. This gives
> (221531 - 221506) = 25 successful requests in 5 secs, so 5r/s, just as
> expected. So far so good.
> 
> Now, what happens if I set rate=5000r/s:
> 
> $ ./wrk -t 100 -c 100 -d 5 http://127.0.0.1/limit/test
> Running 5s test @ http://127.0.0.1/limit/test
>   100 threads and 100 connections
>   Thread Stats   Avg      Stdev     Max   +/- Stdev
>     Latency     3.64ms    5.70ms  36.58ms   87.43%
>     Req/Sec   443.50    191.55     0.89k    65.04%
>   210117 requests in 4.99s, 77.38MB read
>   Non-2xx or 3xx responses: 207671
> Requests/sec:  42070.61
> Transfer/sec:     15.49MB
> 
> This time it is (210117 - 207671) = 2446 successful requests in 5 secs,
> which means 490r/s. Ten times lower then expected. 
> 
> I gathered some more figures, showing the number of 200 responses for
> growing value of "rate" parameter.
> 
> rate=***r/s in zone cfg -- number of 200 responses 
> 100 -- 87
> 200 -- 149
> 500 -- 344
> 1000 -- 452
> 10000 -- 468
> 100000 -- 466
> 
> As you can see, the server keeps returning pretty much constant number of
> 200 responses once the "rate" parameter has surpassed 1000.
> 
> I had a glimpse into the module's code, and this part caught my eye:
> https://github.com/nginx/nginx/blob/nginx-1.7/src/http/modules/ngx_http_limit_req_module.c#L402-L414.
> Basically, if consecutive requests hit the server in the same millisecond,
> the "ms = (ngx_msec_int_t) (now - lr->last)" part evaluates to 0, which sets
> "excess" to 1000, which is very likely to be greater than a "burst" value,
> which results in rejecting the request. This could also mean, that only the
> very first request hitting the server in given millisecond would be handled,
> which seems to be in line with the wrk test results, I've presented above.
> 
> Please let me know if this makes sense to you!
> 
[..]

Yes, the module is limited by millisecond timer resolution, but using so high
rate values without the burst parameter is meaningless.

So, just don't do that.

  wbr, Valentin V. Bartenev