Possible limitation of ngx_http_limit_req_module

[jwroblewski]

Hi,

I'm observing an inconsistent behavior of ngx_http_limit_req_module in nginx
1.7.12. The relevant excerpts from my config:

http {
...
    # A fixed string used as a key, to make all requests fall into the same
zone
    limit_req_zone test_zone zone=test_zone:1m rate=5r/s;
...
    server {
    ...
        location /limit {
            root /test
            limit_req zone=test_zone nodelay;
        }
    ...
    }
}

I use wrk to hammer the server for 5 secs:

$ ./wrk -t 100 -c 100 -d 5 http://127.0.0.1/limit/test
Running 5s test @ http://127.0.0.1/limit/test
  100 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.82ms    2.96ms  15.12ms   88.92%
    Req/Sec   469.03    190.97     0.89k    62.05%
  221531 requests in 5.00s, 81.96MB read
  Non-2xx or 3xx responses: 221506
Requests/sec:  44344.69
Transfer/sec:     16.41MB

So, out of 221531 sent requests, 221506 came back with error. This gives
(221531 - 221506) = 25 successful requests in 5 secs, so 5r/s, just as
expected. So far so good.

Now, what happens if I set rate=5000r/s:

$ ./wrk -t 100 -c 100 -d 5 http://127.0.0.1/limit/test
Running 5s test @ http://127.0.0.1/limit/test
  100 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.64ms    5.70ms  36.58ms   87.43%
    Req/Sec   443.50    191.55     0.89k    65.04%
  210117 requests in 4.99s, 77.38MB read
  Non-2xx or 3xx responses: 207671
Requests/sec:  42070.61
Transfer/sec:     15.49MB

This time it is (210117 - 207671) = 2446 successful requests in 5 secs,
which means 490r/s. Ten times lower then expected. 

I gathered some more figures, showing the number of 200 responses for
growing value of "rate" parameter.

rate=***r/s in zone cfg -- number of 200 responses 
100 -- 87
200 -- 149
500 -- 344
1000 -- 452
10000 -- 468
100000 -- 466

As you can see, the server keeps returning pretty much constant number of
200 responses once the "rate" parameter has surpassed 1000.

I had a glimpse into the module's code, and this part caught my eye:
https://github.com/nginx/nginx/blob/nginx-1.7/src/http/modules/ngx_http_limit_req_module.c#L402-L414.
Basically, if consecutive requests hit the server in the same millisecond,
the "ms = (ngx_msec_int_t) (now - lr->last)" part evaluates to 0, which sets
"excess" to 1000, which is very likely to be greater than a "burst" value,
which results in rejecting the request. This could also mean, that only the
very first request hitting the server in given millisecond would be handled,
which seems to be in line with the wrk test results, I've presented above.

Please let me know if this makes sense to you!

Best regards,
Jakub Wroblewski

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,258803,258803#msg-258803