high "Load Average"

[Sessna]

Stefan Parvu Wrote:
-------------------------------------------------------
>     Give a try to netstat:
thanks for advice, looks like very good tool
 
>  2. vmstat: would as well give you an overall picture of cpu/mem 
>     what does it say on your box: vmstat 1 10 ?
> Run this as well
>     on your DB server.
>      check free column
>      check si, so

Here is output from vmstat run on both servers:

DB server

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0  12488 581340 401492 176548    0    0     0    52 10550 9003 23  6 66  5  0
 4  0  12488 581768 401492 176548    0    0     0     0 10519 8365 18  6 76  0  0
 0  2  12488 581520 401492 176552    0    0     0  1716 4211 4028 13  6 62 20  0
 0  1  12488 581196 401492 176564    0    0     0   800 1763 1531  0  0 67 33  0
12  0  12488 579464 401504 176612    0    0    44   252 10519 17321 35 12 45  7  0
 0  0  12488 583124 401504 176548    0    0   144     0 10094 18755 22  9 65  4  0
 7  0  12488 581612 401504 176460    0    0     0     0 10777 8417  9  6 85  0  0
 5  0  12488 581712 401504 176460    0    0     0   308 10426 8411 10  5 85  0  0
 8  0  12488 584316 401504 176464    0    0     0     0 10515 8168 10  4 86  0  0
 0  6  12488 584180 401504 176464    0    0     0  1660 5757 5053  5  4 49 42  0
 0  5  12488 584008 401504 176464    0    0     0  1240 1429 1286  0  1 28 72  0
10  1  12488 582048 401504 176464    0    0     8  1512 4903 4918  7  3 43 47  0
 4  1  12488 582696 401504 176476    0    0     0  1128 9823 8041 19  8 53 20  0

Plenty of free memory and no swap activity at all


and Web server exactly at the same. Load is very high, system significantly slowed down. 
Load average: 48.14, 63.45, 78.10

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 5 21 1165724 542180  97228 261580    0    0     0  3812 12690 10142 11  9 38 41  0
17  2 1165724 543352  97244 261508    0    0     0  5020 7574 5605  4  5 52 38  0
 3  8 1165724 542296  97248 261332    0    0     0  3492 10717 7997  6  7 65 23  0
 1  9 1165724 545036  97248 261160    0    0     0  3908 6850 4321  3  5 61 30  0
 0 14 1165724 544320  97248 260940    0    0     0  3236 6474 4485  5  5 28 62  0
 0 23 1165724 541884  97260 261432    0    0     0  4212 11362 8626 11  8 25 55  0
 0 14 1165724 541740  97268 261436    0    0     0  3968 6058 4598  4  4 41 51  0
 0  2 1165724 540876  97276 261324    0    0     0  5792 6281 4923  4  5 53 38  0
 1  9 1165716 540732  97296 261260   32    0    32  3780 14479 11158 13  9 53 25  0
 1  9 1165716 541632  97296 261316    0    0     0  4248 6083 3842  1  3 71 24  0
 3 33 1165716 541016  97296 261372    0    0     4  3696 7933 5780  6  9 37 49  0
 2 29 1165716 541864  97300 261488    0    0     4  4128 6081 4607  6  5 40 50  0
 1 45 1165716 538448  97308 261520    0    0     0  3648 12608 10577 11  7 27 56  0
 5 15 1165716 541024  97308 261124    0    0     0  3844 12440 9727 10  9 34 47  0

We have about 540Mb free memory and small si activity (there is only one 32K page reading presented in snipped, but actually I've seen several more 32K reads). As expected plenty of processes in "uninterraptible sleep" state. Looks like enough free resourses (mem, CPU), but system is very slow. I am beginning to think that processes in "uninterruptible sleep" state are driving system to have scheduling lag. Looks for me like they do not consume CPU, but slows down task scheduler. Can it be an issue?
 
Thanks,
Sessna

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,63176,63932#msg-63932