high i/o usage

[Payam Chychi]

On Mon, Oct 5, 2009 at 12:02 AM, Stefanita Rares Dumitrescu
<katmai at keptprivate.com> wrote:
> hi guys,
>
> i am using centos 5, nginx version: nginx/0.8.16
> , and php-fpm and i have switched to nginx on one of my very busy servers.
>
> however, i see very high i/o usage, and the site works pretty sloppy.
>
> below is my iostat and df output. as you can see i get around 25 mb/s
> written at any time.
>
> mysql is on a second drive, and there is pretty much nothing else. however i
> cannot understand why the i/o is so high. i expect that maybe because of log
> writing, but i checked the log folder, and it's not.
>
> I have setup in the beginnging the worker processes to 8, but for some
> unknown reason after 1 2 minutes, nginx stops, so i had to increase it to
> 64, and now it stays up. but i have left the i/o thing, which i cannot make
> anything out of it.
>
>
> Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
> sda             123.88         0.00        25.94          0         52
> sda1              0.00         0.00         0.00          0          0
> sda2              0.00         0.00         0.00          0          0
> sda3              1.00         0.00         0.02          0          0
> sda4            122.89         0.00        25.92          0         52
> sdb              15.42         0.07         0.00          0          0
> sdb1             15.42         0.07         0.00          0          0
> sdc              69.15         0.58         1.27          1          2
> sdc1             69.15         0.58         1.27          1          2
>
>
> root at rum [/usr/local/nginx/logs]# df -h
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/sda4             331G   50G  265G  16% /
> /dev/sda3              15G  1.2G   13G   9% /tmp
> /dev/sda1              99M   18M   77M  19% /boot
> tmpfs                 7.9G     0  7.9G   0% /dev/shm
> /dev/sdb1             367G   60G  289G  18% /home2
> /dev/sdc1              58G   13G   43G  23% /home3
>
>

Hi,

This is a known issue with cent os 5.*, its the HD read/write method
used as default (though i believe that 5.* has major issues)
You will experience the same issue with any utility or tool that write
to disk (try using 'wget' to download a file to disk .. then use wget
to write to /dev/null  'wget -S http://site.com/file -O /dev/null'
you will see the issue. I would also recommend using "dstat" to
diagnose the problem as it will provide you with live data, something
like what you have posted above only you can monitor entire system
i/o,network,cpu resource utilization.

id recommend moving off of centOS (debain/gentoo or a better os,
ubuntu) or perhaps downgrading to 4.*
good luck,
-- 
Payam Tarverdyan Chychi
Network Security Specialist / Network Engineer