Race condition in limit_zone/limit_conn?

Barry Jaspan barry.jaspan at acquia.com
Mon May 3 23:37:49 MSD 2010


I am seeing a behavior in nginx/0.7.62 that I cannot explain.

I want to limit the maximum number of concurrent connections per
virtual host (server block). As an approximation of that, I'm using

    limit_zone max_conn $host 10m;

(10MB because we have a *lot* of domain names) and then within each server block

  limit_conn max_conn N;

to limit to N concurrent connections. To test this, I put a script
called sleep.php on the back-end web nodes that is just

<?php sleep($_GET['sleep']); ?>

Suppose I set N to 3. If I run

ab -n 3 -c 3 'http://domain.name/sleep.php?sleep=30'

it works as expected. All three connections are established and hang
for 30 seconds. During that time, if I try to connect to that host
name in another window, I get a 503. After 30 seconds, the 3 sleeping
requests complete normally.

With N set to 3, if I run

ab -n 4 -c 4 'http://test3.bjaspan.acquia-sites.com/sleep.php?sleep=30'

what I would expect to see is that 3 connections would succeed and the
fourth would fail quickly with a 503. Instead, what happens is that
all 4 fail immediately with a 503.

The log is not particularly illuminating (at least to me):

root at bal-11:~# tail -f /var/log/nginx/error.log | grep "limit"
2010/05/03 19:29:34 [debug] 4080#0: *1 limit zone: 265A3FCB 1
2010/05/03 19:29:34 [debug] 4080#0: *2 limit zone: 265A3FCB 2
2010/05/03 19:29:34 [debug] 4080#0: *4 limit zone: 265A3FCB 3
2010/05/03 19:29:34 [error] 4080#0: *6 limiting connections by zone
"max_conn", client: 127.0.0.1, server: domain.name, request: "GET
/sleep.php?sleep=15 HTTP/1.0", host: "domain.name"
2010/05/03 19:29:34 [debug] 4080#0: *6 http write filter limit 0
2010/05/03 19:29:34 [error] 4080#0: *8 limiting connections by zone
"max_conn", client: 127.0.0.1, server: domain.name, request: "GET
/sleep.php?sleep=15 HTTP/1.0", host: "domain.name"
2010/05/03 19:29:34 [debug] 4080#0: *8 http write filter limit 0
2010/05/03 19:29:35 [error] 4080#0: *9 limiting connections by zone
"max_conn", client: 127.0.0.1, server: domain.name, request: "GET
/sleep.php?sleep=15 HTTP/1.0", host: "domain.name"
2010/05/03 19:29:35 [debug] 4080#0: *9 http write filter limit 0
2010/05/03 19:29:35 [error] 4080#0: *10 limiting connections by zone
"max_conn", client: 127.0.0.1, server: domain.name, request: "GET
/sleep.php?sleep=15 HTTP/1.0", host: "domain.name"
2010/05/03 19:29:35 [debug] 4080#0: *10 http write filter limit 0
2010/05/03 19:29:35 [debug] 4080#0: limit zone cleanup: 265A3FCB 3
2010/05/03 19:29:35 [debug] 4080#0: limit zone cleanup: 265A3FCB 2
2010/05/03 19:29:35 [debug] 4080#0: limit zone cleanup: 265A3FCB 1

Looking in ngx_http_limit_zone_handler(), it seems that only a single
connection should get NGX_HTTP_SERVICE_UNAVAILABLE when the limit is
exceeded, unless I do not understand what the return value from the
handler means (which is entirely possible, I've never looked inside
nginx's code before).

Any thoughts?

Thanks,

Barry



More information about the nginx mailing list