high "Load Average"

[Rob Mueller]

> procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
> r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id 
> wa st
> 5 21 1165724 542180  97228 261580    0    0     0  3812 12690 10142 11  9 
> 38 41  0
> 17  2 1165724 543352  97244 261508    0    0     0  5020 7574 5605  4  5 
> 52 38  0
> 3  8 1165724 542296  97248 261332    0    0     0  3492 10717 7997  6  7 
> 65 23  0
> 1  9 1165724 545036  97248 261160    0    0     0  3908 6850 4321  3  5 61 
> 30  0
> 0 14 1165724 544320  97248 260940    0    0     0  3236 6474 4485  5  5 28 
> 62  0
> 0 23 1165724 541884  97260 261432    0    0     0  4212 11362 8626 11  8 
> 25 55  0
> 0 14 1165724 541740  97268 261436    0    0     0  3968 6058 4598  4  4 41 
> 51  0
> 0  2 1165724 540876  97276 261324    0    0     0  5792 6281 4923  4  5 53 
> 38  0
> 1  9 1165716 540732  97296 261260   32    0    32  3780 14479 11158 13  9 
> 53 25  0
> 1  9 1165716 541632  97296 261316    0    0     0  4248 6083 3842  1  3 71 
> 24  0
> 3 33 1165716 541016  97296 261372    0    0     4  3696 7933 5780  6  9 37 
> 49  0
> 2 29 1165716 541864  97300 261488    0    0     4  4128 6081 4607  6  5 40 
> 50  0
> 1 45 1165716 538448  97308 261520    0    0     0  3648 12608 10577 11  7 
> 27 56  0
> 5 15 1165716 541024  97308 261124    0    0     0  3844 12440 9727 10  9 
> 34 47  0

I don't think there's any network problems, it looks much more like your 
system is running out of disk IO bandwidth. There's a continuous stream of 
data being sent to disk (bo). The actual rate (~4000 blocks/s) isn't that 
high if it's purely linear write, but my *guess* is you're getting quite a 
bit of random IO going on. That's causing lots of processes to block.

What's probably also happening is that your sshd and vpn server are being 
swapped out (swapped = 1.1G, free = 540M), so when you try and connect, they 
have to be swapped back in. Because of all the IO being pushed to disk, the 
reads are starved, and swapping them in takes a while, causing your 
connections to time out.

Basically the question to answer, why and what are your php processing 
writing so much to local disk continuously for?

One thing that can help.

echo 1 > /proc/sys/vm/block_dump; sleep 10; echo 0 > /proc/sys/vm/block_dump

Then look at:

dmesg | less

And scroll down till you see stuff like:

[20394434.668776] pdflush(320): WRITE block 15744 on sda5
[20394434.668783] pdflush(320): WRITE block 15752 on sda5
[20394434.689927] pdflush(320): WRITE block 15760 on sda5
[20394434.699997] pdflush(320): WRITE block 173474328 on sda5
[20394434.772645] cleanup(32657): dirtied inode 4909405 (55FD14AE95D) on 
sda5
[20394434.772674] cleanup(32657): dirtied inode 4909405 (55FD14AE95D) on 
sda5
[20394435.567579] nginx(14952): dirtied inode 3215421100 (0000623907) on 
tmpfs
[20394435.603644] kjournald(2281): WRITE block 901856 on sda1
[20394435.603687] kjournald(2281): WRITE block 901864 on sda1
[20394435.603696] kjournald(2281): WRITE block 901872 on sda1

And see if you can find out what files the php processes are writing to.

Rob