Race condition in limit_zone/limit_conn?

[Barry Jaspan]

Maxim,

The backend proxy server in my example is Apache on another box; there
is no proxy loop. I verified this with the X-Loop-Test header you
suggested.

At your suggestion I started looking more closely into the nginx log,
and I seem to have discovered that the problem is a bug with ab on
MacOS X. nginx is reporting that the remote client is closing the
connections (status 499) in addition to (as you pointed out) creating
extra connections I do not seem to have requested. When I run ab on
Ubuntu (from the machine running nginx itself and from a different
machine), I get the expected behavior: with limit_conn set to 3 and
making 4 concurrent requests, 3 requests succeed, and the 4th gets a
503. And the nginx error log looks much more like what I'd expect:

ab  -c 4 -n 4 'http://maxcon1.bjaspan.acquia-sites.com/sleep.php?sleep=3'

root at bal-11:~# grep limit /var/log/nginx/error.log
2010/05/04 19:05:21 [notice] 936#0: getrlimit(RLIMIT_NOFILE): 10000:10000
2010/05/04 19:05:28 [debug] 938#0: *1 limit zone: 265A3FCB 1
2010/05/04 19:05:28 [debug] 938#0: *2 limit zone: 265A3FCB 2
2010/05/04 19:05:28 [debug] 938#0: *4 limit zone: 265A3FCB 3
2010/05/04 19:05:28 [error] 938#0: *6 limiting connections by zone
"max_conn", client: 174.129.169.92, server: _, request: "GET
/sleep.php?sleep=3 HTTP/1.0", host: "maxcon1.bjaspan.acquia-sites.com"
2010/05/04 19:05:28 [debug] 938#0: *6 http write filter limit 0
2010/05/04 19:05:32 [debug] 938#0: *1 http write filter limit 0
2010/05/04 19:05:32 [debug] 938#0: limit zone cleanup: 265A3FCB 3
2010/05/04 19:05:32 [debug] 938#0: *4 http write filter limit 0
2010/05/04 19:05:32 [debug] 938#0: limit zone cleanup: 265A3FCB 2
2010/05/04 19:05:32 [debug] 938#0: *2 http write filter limit 0
2010/05/04 19:05:32 [debug] 938#0: limit zone cleanup: 265A3FCB 1

So, there is no apparent bug in nginx or my config files, and there
will be no more using ab on MacOS X for me!

Thanks,

Barry


On Tue, May 4, 2010 at 1:27 PM, Maxim Dounin <mdounin at mdounin.ru> wrote:
> Hello!
>
> On Tue, May 04, 2010 at 12:10:26PM -0400, Barry Jaspan wrote:
>
>> Maxim,
>>
>> Thank you for your reply.
>>
>> On Mon, May 3, 2010 at 7:58 PM, Maxim Dounin <mdounin at mdounin.ru> wrote:
>> > You had at least 7 connections (*1, *2, *4, *6, *8, *9, *10)
>> > during the test.  First 3 were passed through, others were
>> > rejected. ... Most likely you configured nginx to proxy to itself and this
>> > causes extra connections.
>>
>> I understand how making nginx proxy to itself would cause extra
>> connections and how those might make all of the initial connections
>> fail. If the limit is 3, I open 3, then each of those 3 attempt to
>> proxy to nginx again, nginx will return 503 to the proxy request which
>> will then be returned to the original requests.
>>
>> However, I do not see how that is happening in my case.
>
> Sure, but 1) you haven't provided any evidence it doesn't happen
> with 3 requests and 2) even assumming it doesn't happen - nobody
> says loop is unconditional.
>
>> I have reduced
>> my configuration file to a minimum. Here it is:
>>
>> ---- snip snip ----
>> user              nginx;
>> worker_processes  10;
>> error_log         /var/log/nginx/error.log debug;
>> pid               /var/run/nginx.pid;
>> events {
>>     worker_connections  2048;
>> }
>> http {
>>   proxy_set_header Host $host;
>>   limit_zone max_conn $host 10m;
>>   limit_conn max_conn 3;
>>   server {
>>     listen       80;
>>     server_name _;
>>     location / {
>>       proxy_pass http://10.252.86.98;
>>     }
>>   }
>> }
>> ---- snip snip ----
>>
>> Does this config file make nginx proxy to itself? I do not see how.
>
> The question is: what handles connections on 10.252.86.98:80 and
> how it handles them?  In most simple case it's the same nginx, and
> loop is clear enough.
>
> [...]
>
>> Any thoughts?
>
> Your logs clearly show there are more requests than you expect
> from ab.  And your config doesn't make proxy loop impossible.
>
> You may either remove "proxy_set_header Host $host" to make sure
> Host header in original request will differ from one in proxied,
> or add something like
>
>    proxy_set_header X-Loop-Test washere;
>
> to make sure requests will be at least distingushable in debug
> log.
>
> Maxim Dounin
>
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://nginx.org/mailman/listinfo/nginx
>