Race condition in limit_zone/limit_conn?

[Barry Jaspan]

I am seeing a behavior in nginx/0.7.62 that I cannot explain.

I want to limit the maximum number of concurrent connections per
virtual host (server block). As an approximation of that, I'm using

    limit_zone max_conn $host 10m;

(10MB because we have a *lot* of domain names) and then within each server block

  limit_conn max_conn N;

to limit to N concurrent connections. To test this, I put a script
called sleep.php on the back-end web nodes that is just

<?php sleep($_GET['sleep']); ?>

Suppose I set N to 3. If I run

ab -n 3 -c 3 'http://domain.name/sleep.php?sleep=30'

it works as expected. All three connections are established and hang
for 30 seconds. During that time, if I try to connect to that host
name in another window, I get a 503. After 30 seconds, the 3 sleeping
requests complete normally.

With N set to 3, if I run

ab -n 4 -c 4 'http://test3.bjaspan.acquia-sites.com/sleep.php?sleep=30'

what I would expect to see is that 3 connections would succeed and the
fourth would fail quickly with a 503. Instead, what happens is that
all 4 fail immediately with a 503.

The log is not particularly illuminating (at least to me):

root at bal-11:~# tail -f /var/log/nginx/error.log | grep "limit"
2010/05/03 19:29:34 [debug] 4080#0: *1 limit zone: 265A3FCB 1
2010/05/03 19:29:34 [debug] 4080#0: *2 limit zone: 265A3FCB 2
2010/05/03 19:29:34 [debug] 4080#0: *4 limit zone: 265A3FCB 3
2010/05/03 19:29:34 [error] 4080#0: *6 limiting connections by zone
"max_conn", client: 127.0.0.1, server: domain.name, request: "GET
/sleep.php?sleep=15 HTTP/1.0", host: "domain.name"
2010/05/03 19:29:34 [debug] 4080#0: *6 http write filter limit 0
2010/05/03 19:29:34 [error] 4080#0: *8 limiting connections by zone
"max_conn", client: 127.0.0.1, server: domain.name, request: "GET
/sleep.php?sleep=15 HTTP/1.0", host: "domain.name"
2010/05/03 19:29:34 [debug] 4080#0: *8 http write filter limit 0
2010/05/03 19:29:35 [error] 4080#0: *9 limiting connections by zone
"max_conn", client: 127.0.0.1, server: domain.name, request: "GET
/sleep.php?sleep=15 HTTP/1.0", host: "domain.name"
2010/05/03 19:29:35 [debug] 4080#0: *9 http write filter limit 0
2010/05/03 19:29:35 [error] 4080#0: *10 limiting connections by zone
"max_conn", client: 127.0.0.1, server: domain.name, request: "GET
/sleep.php?sleep=15 HTTP/1.0", host: "domain.name"
2010/05/03 19:29:35 [debug] 4080#0: *10 http write filter limit 0
2010/05/03 19:29:35 [debug] 4080#0: limit zone cleanup: 265A3FCB 3
2010/05/03 19:29:35 [debug] 4080#0: limit zone cleanup: 265A3FCB 2
2010/05/03 19:29:35 [debug] 4080#0: limit zone cleanup: 265A3FCB 1

Looking in ngx_http_limit_zone_handler(), it seems that only a single
connection should get NGX_HTTP_SERVICE_UNAVAILABLE when the limit is
exceeded, unless I do not understand what the return value from the
handler means (which is entirely possible, I've never looked inside
nginx's code before).

Any thoughts?

Thanks,

Barry