much iowait - how to reduce?

Wed Jun 18 00:11:55 MSD 2008

----- Original Message ----- 
From: "Igor Sysoev" <is at rambler-co.ru>
To: <nginx at sysoev.ru>
Sent: Tuesday, June 17, 2008 8:38 PM
Subject: Re: much iowait - how to reduce?

> On Tue, Jun 17, 2008 at 08:05:10PM +0200, Edo Frederix wrote:
>
>>
>> ----- Original Message ----- 
>> From: "Igor Sysoev" <is at rambler-co.ru>
>> To: <nginx at sysoev.ru>
>> Sent: Tuesday, June 17, 2008 12:40 PM
>> Subject: Re: much iowait - how to reduce?
>>
>>
>> >On Tue, Jun 17, 2008 at 12:16:03PM +0200, Edo Frederix wrote:
>> >
>> >>
>> >>----- Original Message ----- 
>> >>From: "Igor Sysoev" <is at rambler-co.ru>
>> >>To: <nginx at sysoev.ru>
>> >>Sent: Tuesday, June 17, 2008 10:34 AM
>> >>Subject: Re: much iowait - how to reduce?
>> >>
>> >>
>> >>>On Mon, Jun 16, 2008 at 09:15:44PM +0200, Edo Frederix wrote:
>> >>>
>> >>>>I am running nginx/0.7.1 on my debian (2.6.24.3 kernel) system. You 
>> >>>>can
>> >>>>see my nginx.conf here: http://pastebin.com/m63c18e1
>> >>>>
>> >>>>My system has much iowait: http://i27.tinypic.com/288crwm.jpg. This 
>> >>>>is
>> >>>>due to some disk activity every 5 seconds. Here are some samples of
>> >>>>"vmstat 1":
>> >>>>
>> >>>>procs -----------memory---------- ---swap-- -----io---- -system-- 
>> >>>>----cpu----
>> >>>> 0  2      0 3188220  41632  61408    0    0     0  2280 9119 6100  1
>> >>>> 4
>> >>>>73 22
>> >>>> 0  0      0 3187680  41632  61936    0    0     0   272 10516 8027 
>> >>>> 1
>> >>>> 4
>> >>>>93  2
>> >>>> 0  0      0 3188004  41632  61256    0    0     0     0 9446 6479  1
>> >>>> 4
>> >>>>95  0
>> >>>> 0  0      0 3188052  41632  61216    0    0     0     0 9533 6893  1
>> >>>> 3
>> >>>>96  0
>> >>>> 0  0      0 3187880  41632  62052    0    0     0     0 10639 8117 
>> >>>> 2
>> >>>> 6
>> >>>>93  0
>> >>>> 0  0      0 3186936  41632  64132    0    0     0     0 9797 6898  1
>> >>>> 4
>> >>>>95  0
>> >>>> 1  1      0 3189304  41636  61324    0    0     0  4692 9052 6414  1
>> >>>> 4
>> >>>>65 29
>> >>>> 0  3      0 3188508  41636  62656    0    0     0  1124 9107 6302  2
>> >>>> 4
>> >>>>72 22
>> >>>> 0  0      0 3187448  41636  63044    0    0     0     0 8601 5591  1
>> >>>> 5
>> >>>>87  8
>> >>>> 0  0      0 3184968  41636  65344    0    0     0     0 10273 7832 
>> >>>> 2
>> >>>> 6
>> >>>>92  0
>> >>>> 0  0      0 3184900  41636  65024    0    0     0     0 10104 7827 
>> >>>> 2
>> >>>> 5
>> >>>>93  0
>> >>>> 1  0      0 3181960  41636  68428    0    0     0     0 9675 7242  2
>> >>>> 5
>> >>>>93  0
>> >>>> 0  4      0 3183416  41636  68892    0    0     0  9528 7025 3921  1
>> >>>> 2
>> >>>>55 42
>> >>>> 0  0      0 3186076  41636  63620    0    0     0   440 8508 5839  1
>> >>>> 5
>> >>>>60 35
>> >>>>
>> >>>>As you can see in my nginx.conf, I have disabled logging (even error
>> >>>>log). I can not determine where my disk is actually writing, but what 
>> >>>>I
>> >>>>do know is that it is related to nginx. When our site gets busy
>> >>>>(200mbit
>> >>>>traffic), iowait increases. There are no other important processes
>> >>>>running on the server.
>> >>>>
>> >>>>My question now is: Is this normal behaviour and how can I reduce the
>> >>>>iowait?
>> >>>
>> >>>If you serve large static content, then this is normal: nginx worker
>> >>>processes wait on disk reads.
>> >>>
>> >>I am not running large static content, only some small images and some
>> >>php
>> >>generated html from my backends. Besides, the disk is only writing, not
>> >>reading:
>> >>
>> >>Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
>> >>sda               8.00         0.00      1168.00          0       1168
>> >>
>> >>Barry from wordpress.com told me that adding the line
>> >>"client_body_temp_path /dev/shm;" would help, because of this quote: 
>> >>"If
>> >>the request body is more than the buffer, then the entire request body 
>> >>or
>> >>some part is written in a temporary file."
>> >>(http://wiki.codemongers.com/NginxHttpCoreModule#client_body_buffer_size).
>> >>Now this client body should get written to some shared memory, not to 
>> >>the
>> >>disk.
>> >>
>> >>Doesn't seem to work though..
>> >
>> >Do you have many uploads ?
>> >
>> >Do you use 32-bit or 64-bit OS ?
>> >Try to increase number of proxy_buffers:
>> >
>> >     proxy_buffer_size      4k;
>> >     proxy_buffers          64  4k;
>> >
>> >Then backend responses up to 260K will be buffered in memory.
>> >
>>
>> No uploads. All incomming data that gets stored is handled by the backend
>> webservers, and probably all send to an independent MySQL server. I have 
>> a
>> 32bit OS.
>>
>> I have been tuning a bit with proxy_buffer_size and proxy_buffers. Turns
>> out that these settings reduce the iowait drastically for me:
>>
>> proxy_buffer_size 32k;
>> proxy_buffers  512 32k;
>>
>> I did a benchmark, and the server went to 20% CPU utilization, from which
>> maybe 1% was iowait - that used to be 15% iowait. Tonight it will get 
>> busy
>> on the server, so let's see if we will have a nice proof of concept.
>
> 512*32k+32k is up to 16M per request. It's too big. What is typical size
> of your responses ? It's better to set proxy_buffers to conform them
> and allow to write large responses to a disk.
>
>
> -- 
> Igor Sysoev
> http://sysoev.ru/en/
>

The thing is that I have some files that I publish on the website. These 
files are about 3mb large. A typical size of a response would probably be 
around 30kb. On my server I'm running with 4GB memory, so there is free 
memory enough to run all buffers from the memory. I experienced that iowait 
would drop if I increased the number of buffers.

Given this info, what would you recommend?

E. 