nginx serving large files - performance issues with more than ~800-1000 connections

[Tomasz Chmielewski]

On 05/24/2012 06:25 PM, Maxim Dounin wrote:

>> Does it suggest nginx issues? Because the second nginx instance
>> serves the files fine.
>>
>> Or maybe some system / sysctl parameters?
> 
> It suggests you are disk-bound and all nginx workers are busy
> waiting for I/O operations.  Try looking here for basic
> optimization steps:
> 
> http://mailman.nginx.org/pipermail/nginx/2012-May/033761.html

I've tried to follow these recommendations, but don't really see any improvement.

The systems are not disk bound (see below).

Even if I try to fetch a file which is stored in tmpfs, it is slow - 20, 30 secs, even more, like here:

[root at da1 ~]# time curl ca3/404/404.html
curl: (7) couldn't connect to host

real    1m3.204s
user    0m0.000s
sys     0m0.000s


I see it only when the number of established connections to nginx is around 700 and more (most serving large files, so the connections are long-lived).



Disk load:

root at ca2:~# iostat -k 1
(...)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          21.23    0.00   25.69    2.41    0.00   50.66

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda               0.00         0.00         0.00          0          0
sdb              26.00      1240.00         0.00       1240          0
sdc              23.00      2324.00         0.00       2324          0
sdd              12.00       916.00         0.00        916          0
sde               9.00       532.00         0.00        532          0
sdf               9.00       532.00         0.00        532          0
sdg               9.00       532.00         0.00        532          0
sdh              22.00      2196.00         0.00       2196          0
sdi              32.00      1044.00       172.00       1044        172
sdj               5.00        20.00         0.00         20          0
sdk              13.00      1044.00         0.00       1044          0
sdl              17.00      1556.00         0.00       1556          0
sdm              18.00       940.00         0.00        940          0


root at ca2:~# iostat -x 1

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          21.81    0.00   29.46    4.46    0.00   44.26

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.00    26.00    6.00    2.00  1536.00   224.00   220.00     0.00    0.00   0.00   0.00
sdc               0.00     0.00   18.00    0.00  2656.00     0.00   147.56     0.04    2.22   1.11   2.00
sdd               0.00    33.00    4.00    3.00  1024.00   288.00   187.43     0.00    0.00   0.00   0.00
sde               0.00    26.00    4.00    2.00   768.00   224.00   165.33     0.00    0.00   0.00   0.00
sdf               0.00    23.00    1.00    2.00   192.00   200.00   130.67     0.02    6.67   6.67   2.00
sdg               0.00    27.00    6.00    2.00   560.00   232.00    99.00     0.09   11.25   2.50   2.00
sdh               0.00    39.00   28.00   19.00  2392.00   464.00    60.77     0.24    5.11   0.85   4.00
sdi               0.00     0.00    8.00    0.00  1792.00     0.00   224.00     0.02    2.50   1.25   1.00
sdj               0.00    25.00    0.00    2.00     0.00   216.00   108.00     0.00    0.00   0.00   0.00
sdk               0.00     0.00    8.00    0.00  2048.00     0.00   256.00     0.03    3.75   2.50   2.00
sdl               0.00    28.00   14.00    2.00  1152.00   240.00    87.00     0.00    0.00   0.00   0.00
sdm               0.00    60.00    4.00    3.00  1024.00   504.00   218.29     0.01    1.43   1.43   1.00



-- 
Tomasz Chmielewski
http://www.ptraveler.com