much iowait - how to reduce?

[Edo Frederix]

----- Original Message ----- 
From: "Igor Sysoev" <is at rambler-co.ru>
To: <nginx at sysoev.ru>
Sent: Tuesday, June 17, 2008 12:40 PM
Subject: Re: much iowait - how to reduce?


> On Tue, Jun 17, 2008 at 12:16:03PM +0200, Edo Frederix wrote:
>
>>
>> ----- Original Message ----- 
>> From: "Igor Sysoev" <is at rambler-co.ru>
>> To: <nginx at sysoev.ru>
>> Sent: Tuesday, June 17, 2008 10:34 AM
>> Subject: Re: much iowait - how to reduce?
>>
>>
>> >On Mon, Jun 16, 2008 at 09:15:44PM +0200, Edo Frederix wrote:
>> >
>> >>I am running nginx/0.7.1 on my debian (2.6.24.3 kernel) system. You can
>> >>see my nginx.conf here: http://pastebin.com/m63c18e1
>> >>
>> >>My system has much iowait: http://i27.tinypic.com/288crwm.jpg. This is
>> >>due to some disk activity every 5 seconds. Here are some samples of
>> >>"vmstat 1":
>> >>
>> >>procs -----------memory---------- ---swap-- -----io---- -system-- 
>> >>----cpu----
>> >> 0  2      0 3188220  41632  61408    0    0     0  2280 9119 6100  1 
>> >> 4
>> >>73 22
>> >> 0  0      0 3187680  41632  61936    0    0     0   272 10516 8027  1 
>> >> 4
>> >>93  2
>> >> 0  0      0 3188004  41632  61256    0    0     0     0 9446 6479  1 
>> >> 4
>> >>95  0
>> >> 0  0      0 3188052  41632  61216    0    0     0     0 9533 6893  1 
>> >> 3
>> >>96  0
>> >> 0  0      0 3187880  41632  62052    0    0     0     0 10639 8117  2 
>> >> 6
>> >>93  0
>> >> 0  0      0 3186936  41632  64132    0    0     0     0 9797 6898  1 
>> >> 4
>> >>95  0
>> >> 1  1      0 3189304  41636  61324    0    0     0  4692 9052 6414  1 
>> >> 4
>> >>65 29
>> >> 0  3      0 3188508  41636  62656    0    0     0  1124 9107 6302  2 
>> >> 4
>> >>72 22
>> >> 0  0      0 3187448  41636  63044    0    0     0     0 8601 5591  1 
>> >> 5
>> >>87  8
>> >> 0  0      0 3184968  41636  65344    0    0     0     0 10273 7832  2 
>> >> 6
>> >>92  0
>> >> 0  0      0 3184900  41636  65024    0    0     0     0 10104 7827  2 
>> >> 5
>> >>93  0
>> >> 1  0      0 3181960  41636  68428    0    0     0     0 9675 7242  2 
>> >> 5
>> >>93  0
>> >> 0  4      0 3183416  41636  68892    0    0     0  9528 7025 3921  1 
>> >> 2
>> >>55 42
>> >> 0  0      0 3186076  41636  63620    0    0     0   440 8508 5839  1 
>> >> 5
>> >>60 35
>> >>
>> >>As you can see in my nginx.conf, I have disabled logging (even error
>> >>log). I can not determine where my disk is actually writing, but what I
>> >>do know is that it is related to nginx. When our site gets busy 
>> >>(200mbit
>> >>traffic), iowait increases. There are no other important processes
>> >>running on the server.
>> >>
>> >>My question now is: Is this normal behaviour and how can I reduce the
>> >>iowait?
>> >
>> >If you serve large static content, then this is normal: nginx worker
>> >processes wait on disk reads.
>> >
>> I am not running large static content, only some small images and some 
>> php
>> generated html from my backends. Besides, the disk is only writing, not
>> reading:
>>
>> Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
>> sda               8.00         0.00      1168.00          0       1168
>>
>> Barry from wordpress.com told me that adding the line
>> "client_body_temp_path /dev/shm;" would help, because of this quote:  "If
>> the request body is more than the buffer, then the entire request body or
>> some part is written in a temporary file."
>> (http://wiki.codemongers.com/NginxHttpCoreModule#client_body_buffer_size).
>> Now this client body should get written to some shared memory, not to the
>> disk.
>>
>> Doesn't seem to work though..
>
> Do you have many uploads ?
>
> Do you use 32-bit or 64-bit OS ?
> Try to increase number of proxy_buffers:
>
>      proxy_buffer_size      4k;
>      proxy_buffers          64  4k;
>
> Then backend responses up to 260K will be buffered in memory.
>
>
> -- 
> Igor Sysoev
> http://sysoev.ru/en/
>

No uploads. All incomming data that gets stored is handled by the backend 
webservers, and probably all send to an independent MySQL server. I have a 
32bit OS.

I have been tuning a bit with proxy_buffer_size and proxy_buffers. Turns out 
that these settings reduce the iowait drastically for me:

proxy_buffer_size 32k;
proxy_buffers  512 32k;

I did a benchmark, and the server went to 20% CPU utilization, from which 
maybe 1% was iowait - that used to be 15% iowait. Tonight it will get busy 
on the server, so let's see if we will have a nice proof of concept.

Anyhow, thanks for your time Igor, very much appreciated.

E.