much iowait - how to reduce?

[Edo Frederix]

----- Original Message ----- 
From: "Igor Sysoev" <is at rambler-co.ru>
To: <nginx at sysoev.ru>
Sent: Tuesday, June 17, 2008 10:34 AM
Subject: Re: much iowait - how to reduce?


> On Mon, Jun 16, 2008 at 09:15:44PM +0200, Edo Frederix wrote:
>
>> I am running nginx/0.7.1 on my debian (2.6.24.3 kernel) system. You can 
>> see my nginx.conf here: http://pastebin.com/m63c18e1
>>
>> My system has much iowait: http://i27.tinypic.com/288crwm.jpg. This is 
>> due to some disk activity every 5 seconds. Here are some samples of 
>> "vmstat 1":
>>
>> procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
>>  0  2      0 3188220  41632  61408    0    0     0  2280 9119 6100  1  4 
>> 73 22
>>  0  0      0 3187680  41632  61936    0    0     0   272 10516 8027  1  4 
>> 93  2
>>  0  0      0 3188004  41632  61256    0    0     0     0 9446 6479  1  4 
>> 95  0
>>  0  0      0 3188052  41632  61216    0    0     0     0 9533 6893  1  3 
>> 96  0
>>  0  0      0 3187880  41632  62052    0    0     0     0 10639 8117  2  6 
>> 93  0
>>  0  0      0 3186936  41632  64132    0    0     0     0 9797 6898  1  4 
>> 95  0
>>  1  1      0 3189304  41636  61324    0    0     0  4692 9052 6414  1  4 
>> 65 29
>>  0  3      0 3188508  41636  62656    0    0     0  1124 9107 6302  2  4 
>> 72 22
>>  0  0      0 3187448  41636  63044    0    0     0     0 8601 5591  1  5 
>> 87  8
>>  0  0      0 3184968  41636  65344    0    0     0     0 10273 7832  2  6 
>> 92  0
>>  0  0      0 3184900  41636  65024    0    0     0     0 10104 7827  2  5 
>> 93  0
>>  1  0      0 3181960  41636  68428    0    0     0     0 9675 7242  2  5 
>> 93  0
>>  0  4      0 3183416  41636  68892    0    0     0  9528 7025 3921  1  2 
>> 55 42
>>  0  0      0 3186076  41636  63620    0    0     0   440 8508 5839  1  5 
>> 60 35
>>
>> As you can see in my nginx.conf, I have disabled logging (even error 
>> log). I can not determine where my disk is actually writing, but what I 
>> do know is that it is related to nginx. When our site gets busy (200mbit 
>> traffic), iowait increases. There are no other important processes 
>> running on the server.
>>
>> My question now is: Is this normal behaviour and how can I reduce the 
>> iowait?
>
> If you serve large static content, then this is normal: nginx worker
> processes wait on disk reads.
>
>
> -- 
> Igor Sysoev
> http://sysoev.ru/en/
>

I am not running large static content, only some small images and some php 
generated html from my backends. Besides, the disk is only writing, not 
reading:

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               8.00         0.00      1168.00          0       1168

Barry from wordpress.com told me that adding the line "client_body_temp_path 
/dev/shm;" would help, because of this quote:  "If the request body is more 
than the buffer, then the entire request body or some part is written in a 
temporary file." 
(http://wiki.codemongers.com/NginxHttpCoreModule#client_body_buffer_size). 
Now this client body should get written to some shared memory, not to the 
disk.

Doesn't seem to work though..