much iowait - how to reduce?

[Igor Sysoev]

On Tue, Jun 17, 2008 at 08:05:10PM +0200, Edo Frederix wrote:

> 
> ----- Original Message ----- 
> From: "Igor Sysoev" <is at rambler-co.ru>
> To: <nginx at sysoev.ru>
> Sent: Tuesday, June 17, 2008 12:40 PM
> Subject: Re: much iowait - how to reduce?
> 
> 
> >On Tue, Jun 17, 2008 at 12:16:03PM +0200, Edo Frederix wrote:
> >
> >>
> >>----- Original Message ----- 
> >>From: "Igor Sysoev" <is at rambler-co.ru>
> >>To: <nginx at sysoev.ru>
> >>Sent: Tuesday, June 17, 2008 10:34 AM
> >>Subject: Re: much iowait - how to reduce?
> >>
> >>
> >>>On Mon, Jun 16, 2008 at 09:15:44PM +0200, Edo Frederix wrote:
> >>>
> >>>>I am running nginx/0.7.1 on my debian (2.6.24.3 kernel) system. You can
> >>>>see my nginx.conf here: http://pastebin.com/m63c18e1
> >>>>
> >>>>My system has much iowait: http://i27.tinypic.com/288crwm.jpg. This is
> >>>>due to some disk activity every 5 seconds. Here are some samples of
> >>>>"vmstat 1":
> >>>>
> >>>>procs -----------memory---------- ---swap-- -----io---- -system-- 
> >>>>----cpu----
> >>>> 0  2      0 3188220  41632  61408    0    0     0  2280 9119 6100  1 
> >>>> 4
> >>>>73 22
> >>>> 0  0      0 3187680  41632  61936    0    0     0   272 10516 8027  1 
> >>>> 4
> >>>>93  2
> >>>> 0  0      0 3188004  41632  61256    0    0     0     0 9446 6479  1 
> >>>> 4
> >>>>95  0
> >>>> 0  0      0 3188052  41632  61216    0    0     0     0 9533 6893  1 
> >>>> 3
> >>>>96  0
> >>>> 0  0      0 3187880  41632  62052    0    0     0     0 10639 8117  2 
> >>>> 6
> >>>>93  0
> >>>> 0  0      0 3186936  41632  64132    0    0     0     0 9797 6898  1 
> >>>> 4
> >>>>95  0
> >>>> 1  1      0 3189304  41636  61324    0    0     0  4692 9052 6414  1 
> >>>> 4
> >>>>65 29
> >>>> 0  3      0 3188508  41636  62656    0    0     0  1124 9107 6302  2 
> >>>> 4
> >>>>72 22
> >>>> 0  0      0 3187448  41636  63044    0    0     0     0 8601 5591  1 
> >>>> 5
> >>>>87  8
> >>>> 0  0      0 3184968  41636  65344    0    0     0     0 10273 7832  2 
> >>>> 6
> >>>>92  0
> >>>> 0  0      0 3184900  41636  65024    0    0     0     0 10104 7827  2 
> >>>> 5
> >>>>93  0
> >>>> 1  0      0 3181960  41636  68428    0    0     0     0 9675 7242  2 
> >>>> 5
> >>>>93  0
> >>>> 0  4      0 3183416  41636  68892    0    0     0  9528 7025 3921  1 
> >>>> 2
> >>>>55 42
> >>>> 0  0      0 3186076  41636  63620    0    0     0   440 8508 5839  1 
> >>>> 5
> >>>>60 35
> >>>>
> >>>>As you can see in my nginx.conf, I have disabled logging (even error
> >>>>log). I can not determine where my disk is actually writing, but what I
> >>>>do know is that it is related to nginx. When our site gets busy 
> >>>>(200mbit
> >>>>traffic), iowait increases. There are no other important processes
> >>>>running on the server.
> >>>>
> >>>>My question now is: Is this normal behaviour and how can I reduce the
> >>>>iowait?
> >>>
> >>>If you serve large static content, then this is normal: nginx worker
> >>>processes wait on disk reads.
> >>>
> >>I am not running large static content, only some small images and some 
> >>php
> >>generated html from my backends. Besides, the disk is only writing, not
> >>reading:
> >>
> >>Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
> >>sda               8.00         0.00      1168.00          0       1168
> >>
> >>Barry from wordpress.com told me that adding the line
> >>"client_body_temp_path /dev/shm;" would help, because of this quote:  "If
> >>the request body is more than the buffer, then the entire request body or
> >>some part is written in a temporary file."
> >>(http://wiki.codemongers.com/NginxHttpCoreModule#client_body_buffer_size).
> >>Now this client body should get written to some shared memory, not to the
> >>disk.
> >>
> >>Doesn't seem to work though..
> >
> >Do you have many uploads ?
> >
> >Do you use 32-bit or 64-bit OS ?
> >Try to increase number of proxy_buffers:
> >
> >     proxy_buffer_size      4k;
> >     proxy_buffers          64  4k;
> >
> >Then backend responses up to 260K will be buffered in memory.
> >
>
> No uploads. All incomming data that gets stored is handled by the backend 
> webservers, and probably all send to an independent MySQL server. I have a 
> 32bit OS.
> 
> I have been tuning a bit with proxy_buffer_size and proxy_buffers. Turns 
> out that these settings reduce the iowait drastically for me:
> 
> proxy_buffer_size 32k;
> proxy_buffers  512 32k;
> 
> I did a benchmark, and the server went to 20% CPU utilization, from which 
> maybe 1% was iowait - that used to be 15% iowait. Tonight it will get busy 
> on the server, so let's see if we will have a nice proof of concept.

512*32k+32k is up to 16M per request. It's too big. What is typical size
of your responses ? It's better to set proxy_buffers to conform them
and allow to write large responses to a disk.


-- 
Igor Sysoev
http://sysoev.ru/en/