High response time at high concurrent connections

[laltin]

I have a news site which is run by 4 tornado instances and nginx as reverse
proxy in front of them.

Pages are rendered and cached in memcached so generally the response time is
less than 3 ms according to tornado logs.

[I 130918 18:35:37 web:1462] 200 GET / (***.***.***.**) 2.43ms
[I 130918 18:35:37 web:1462] 200 GET / (***.***.***.**) 3.41ms
[I 130918 18:35:37 web:1462] 200 GET / (***.***.***.**) 1.96ms
[I 130918 18:35:37 web:1462] 200 GET / (***.***.***.**) 2.48ms
[I 130918 18:35:37 web:1462] 200 GET / (***.***.***.**) 4.09ms
[I 130918 18:35:37 web:1462] 200 GET / (***.***.***.**) 2.43ms
[I 130918 18:35:37 web:1462] 200 GET / (***.***.***.**) 2.49ms
[I 130918 18:35:38 web:1462] 200 GET / (***.***.***.**) 2.25ms
[I 130918 18:35:38 web:1462] 200 GET / (***.***.***.**) 2.39ms
[I 130918 18:35:38 web:1462] 200 GET / (***.***.***.**) 1.93ms
[I 130918 18:35:38 web:1462] 200 GET / (***.***.***.**) 1.70ms
[I 130918 18:35:38 web:1462] 200 GET / (***.***.***.**) 2.08ms
[I 130918 18:35:38 web:1462] 200 GET / (***.***.***.**) 1.72ms
[I 130918 18:35:38 web:1462] 200 GET / (***.***.***.**) 2.02ms
[I 130918 18:35:38 web:1462] 200 GET / (***.***.***.**) 1.70ms
[I 130918 18:35:38 web:1462] 200 GET / (***.***.***.**) 1.74ms
[I 130918 18:35:38 web:1462] 200 GET / (***.***.***.**) 1.85ms
[I 130918 18:35:38 web:1462] 200 GET / (***.***.***.**) 1.60ms
[I 130918 18:35:38 web:1462] 200 GET / (***.***.***.**) 1.83ms
[I 130918 18:35:38 web:1462] 200 GET / (***.***.***.**) 2.65ms

When I test this site with ab at concurrency level 1000 I get response times
around 0.8 seconds. Here is the benchmark result:

Document Length:        12036 bytes

Concurrency Level:      1000
Time taken for tests:   7.974 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    10000
Total transferred:      122339941 bytes
HTML transferred:       120549941 bytes
Requests per second:    1254.07 [#/sec] (mean)
Time per request:       797.407 [ms] (mean)
Time per request:       0.797 [ms] (mean, across all concurrent requests)
Transfer rate:          14982.65 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    7  20.8      0      86
Processing:    57  508 473.9    315    7014
Waiting:       57  508 473.9    315    7014
Total:        143  515 471.5    321    7014

Percentage of the requests served within a certain time (ms)
  50%    321
  66%    371
  75%    455
  80%    497
  90%   1306
  95%   1354
  98%   1405
  99%   3009
 100%   7014 (longest request)

I can handle ~1200 requests/seconds with 1000 concurrent connections and
when I do the same benchmark with 100 concurrent connections I can again
handle around 1200 requests/second but response time drops to ~80 ms.

When it comes to real life with 1000 concurrent connections users will face
0.8 seconds response time which I think is a bad value.

My question is why response times increase when concurrency level is
increased?

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,243246,243246#msg-243246