high "Load Average"

[Stefan Parvu]

> >     Give a try to netstat:
> thanks for advice, looks like very good tool
>

I meant nicstat. Works on Linux, Solaris.
Probable worth of considering porting this to FreeBSD too.
  
> and Web server exactly at the same. Load is very high, system significantly slowed down. 
> Load average: 48.14, 63.45, 78.10
> 
> procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
>  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
>  5 21 1165724 542180  97228 261580    0    0     0  3812 12690 10142 11  9 38 41  0
> 17  2 1165724 543352  97244 261508    0    0     0  5020 7574 5605  4  5 52 38  0
>  3  8 1165724 542296  97248 261332    0    0     0  3492 10717 7997  6  7 65 23  0
>  1  9 1165724 545036  97248 261160    0    0     0  3908 6850 4321  3  5 61 30  0
>  0 14 1165724 544320  97248 260940    0    0     0  3236 6474 4485  5  5 28 62  0
>  0 23 1165724 541884  97260 261432    0    0     0  4212 11362 8626 11  8 25 55  0
>  0 14 1165724 541740  97268 261436    0    0     0  3968 6058 4598  4  4 41 51  0
>  0  2 1165724 540876  97276 261324    0    0     0  5792 6281 4923  4  5 53 38  0
>  1  9 1165716 540732  97296 261260   32    0    32  3780 14479 11158 13  9 53 25  0
>  1  9 1165716 541632  97296 261316    0    0     0  4248 6083 3842  1  3 71 24  0
>  3 33 1165716 541016  97296 261372    0    0     4  3696 7933 5780  6  9 37 49  0
>  2 29 1165716 541864  97300 261488    0    0     4  4128 6081 4607  6  5 40 50  0
>  1 45 1165716 538448  97308 261520    0    0     0  3648 12608 10577 11  7 27 56  0
>  5 15 1165716 541024  97308 261124    0    0     0  3844 12440 9727 10  9 34 47  0
>

we have nginx -> php + fastcgi -> db ... others
vmstat tells us that there are some processes waiting for IO 
, b column, and most likely these are the connections mapped to
the fastcgi process. So nginx passed the requests to the 
fcgi process (php-cgi etc...) and then fcgi will have do to
its work.

Could this be a problem of the php/fcgi process(es) running on that
box ? I mean the php-cgi which has nothing to do with the nginx
server. Configuration, settings of the FCGI/PHP module ?

 
> We have about 540Mb free memory and small si activity (there is only one 32K page reading presented in snipped, but actually I've seen several more 32K reads). As expected plenty of processes in "uninterraptible sleep" state. Looks like enough free resourses (mem, CPU), but system is very slow. I am beginning to think that processes in "uninterruptible sleep" state are driving system to have scheduling lag. Looks for me like they do not consume CPU, but slows down task scheduler. Can it be an issue?
>

Good is to have latest patches applied etc. To me this is a good candidate for 
DTrace (Solaris) or SystemTap (linux). Probable check php/cgi settings, nginx
settings for fcgi, linux kernel patches, if any. As well try to experiment with 
a much smaller application first. See if you observe same thing (good to have a 
test env where you could reproduce these sort of things)

stefan