high "Load Average"

[Cliff Wells]

On Sat, 2010-03-13 at 02:29 -0500, Sessna wrote:
> Cliff Wells Wrote:
> -------------------------------------------------------
> > You fail to mention what sort of disk subsystem you are using.   
> One SCSI disk
> 
> > High load is often indicative of I/O bottlenecks.   
> > You also don't mention what OS you are using, 
> CentOS release 5.4
> 
> > you should install iotop and take a look at what process is driving the load
> Here is the snippet from iotop output
> Total DISK READ: 3.33 K/s | Total DISK WRITE: 3.27 M/s
>   TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
> 20264 be/4 nobody      0.00 B/s    0.00 B/s  0.00 % 99.99 % php-cgi --fpm --fpm-config /etc/php-fp
> 21604 be/4 nobody      0.00 B/s    0.00 B/s  0.00 % 99.99 % php-cgi --fpm --fpm-config /etc/php-fp
> 21605 be/4 nobody      0.00 B/s    0.00 B/s  0.00 % 99.99 % php-cgi --fpm --fpm-config /etc/php-fp
> 20054 be/4 nobody      0.00 B/s    0.00 B/s  0.00 % 99.99 % php-cgi --fpm --fpm-config /etc/php-fp
> 20902 be/4 nobody      0.00 B/s    0.00 B/s  0.00 % 99.99 % php-cgi --fpm --fpm-config /etc/php-fp
> 20275 be/4 nobody      0.00 B/s    0.00 B/s  0.00 % 99.99 % php-cgi --fpm --fpm-config /etc/php-fp
> 20828 be/4 nobody      0.00 B/s    0.00 B/s  0.00 % 99.99 % php-cgi --fpm --fpm-config /etc/php-fp
> 
> Looks like php-cgi makes 99.99% IO, but not really sure what this means

This means that php-cgi is spending 99.99% of its time in I/O, not that
it's creating 99.99% of your I/O.   It means php-cgi is waiting almost
all the time.

> > (it might be Nginx, but more likely MySQL - restarting Nginx simply
> cancels whatever pending
> > requests MySQL had is my guess).
> MySQL is running on another PC, so php makes network connections to
> DB. It may take a while, of course, but should not php being blocked
> and sleeping while network system is processing request?

Yes, and this is certainly what is happening.   You have to remember
that "load" is an indication of how many processes are waiting to be
scheduled, and this includes processes that are waiting on I/O.  In this
case it looks like PHP isn't generating a lot of I/O, rather it is
*waiting*, creating a high load, and at the same time, you are running
out of PHP processes to handle new requests, which is why the box slows
down.

I'd follow these steps:

1) check the state of the MySQL server.  See if this machine is
overloaded.

2) check MySQL itself.  Turn on query logging and see if you have some
query that's taking a long time to complete.   

3) take a look at your network to make sure that there isn't an issue
there (saturation, packet loss, etc).

Regards,
Cliff