much iowait - how to reduce?

[Igor Sysoev]

On Tue, Jun 17, 2008 at 12:16:03PM +0200, Edo Frederix wrote:

> 
> ----- Original Message ----- 
> From: "Igor Sysoev" <is at rambler-co.ru>
> To: <nginx at sysoev.ru>
> Sent: Tuesday, June 17, 2008 10:34 AM
> Subject: Re: much iowait - how to reduce?
> 
> 
> >On Mon, Jun 16, 2008 at 09:15:44PM +0200, Edo Frederix wrote:
> >
> >>I am running nginx/0.7.1 on my debian (2.6.24.3 kernel) system. You can 
> >>see my nginx.conf here: http://pastebin.com/m63c18e1
> >>
> >>My system has much iowait: http://i27.tinypic.com/288crwm.jpg. This is 
> >>due to some disk activity every 5 seconds. Here are some samples of 
> >>"vmstat 1":
> >>
> >>procs -----------memory---------- ---swap-- -----io---- -system-- 
> >>----cpu----
> >> 0  2      0 3188220  41632  61408    0    0     0  2280 9119 6100  1  4 
> >>73 22
> >> 0  0      0 3187680  41632  61936    0    0     0   272 10516 8027  1  4 
> >>93  2
> >> 0  0      0 3188004  41632  61256    0    0     0     0 9446 6479  1  4 
> >>95  0
> >> 0  0      0 3188052  41632  61216    0    0     0     0 9533 6893  1  3 
> >>96  0
> >> 0  0      0 3187880  41632  62052    0    0     0     0 10639 8117  2  6 
> >>93  0
> >> 0  0      0 3186936  41632  64132    0    0     0     0 9797 6898  1  4 
> >>95  0
> >> 1  1      0 3189304  41636  61324    0    0     0  4692 9052 6414  1  4 
> >>65 29
> >> 0  3      0 3188508  41636  62656    0    0     0  1124 9107 6302  2  4 
> >>72 22
> >> 0  0      0 3187448  41636  63044    0    0     0     0 8601 5591  1  5 
> >>87  8
> >> 0  0      0 3184968  41636  65344    0    0     0     0 10273 7832  2  6 
> >>92  0
> >> 0  0      0 3184900  41636  65024    0    0     0     0 10104 7827  2  5 
> >>93  0
> >> 1  0      0 3181960  41636  68428    0    0     0     0 9675 7242  2  5 
> >>93  0
> >> 0  4      0 3183416  41636  68892    0    0     0  9528 7025 3921  1  2 
> >>55 42
> >> 0  0      0 3186076  41636  63620    0    0     0   440 8508 5839  1  5 
> >>60 35
> >>
> >>As you can see in my nginx.conf, I have disabled logging (even error 
> >>log). I can not determine where my disk is actually writing, but what I 
> >>do know is that it is related to nginx. When our site gets busy (200mbit 
> >>traffic), iowait increases. There are no other important processes 
> >>running on the server.
> >>
> >>My question now is: Is this normal behaviour and how can I reduce the 
> >>iowait?
> >
> >If you serve large static content, then this is normal: nginx worker
> >processes wait on disk reads.
> >
> I am not running large static content, only some small images and some php 
> generated html from my backends. Besides, the disk is only writing, not 
> reading:
> 
> Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
> sda               8.00         0.00      1168.00          0       1168
> 
> Barry from wordpress.com told me that adding the line 
> "client_body_temp_path /dev/shm;" would help, because of this quote:  "If 
> the request body is more than the buffer, then the entire request body or 
> some part is written in a temporary file." 
> (http://wiki.codemongers.com/NginxHttpCoreModule#client_body_buffer_size). 
> Now this client body should get written to some shared memory, not to the 
> disk.
> 
> Doesn't seem to work though.. 

Do you have many uploads ?

Do you use 32-bit or 64-bit OS ?
Try to increase number of proxy_buffers:

      proxy_buffer_size      4k;
      proxy_buffers          64  4k;

Then backend responses up to 260K will be buffered in memory.


-- 
Igor Sysoev
http://sysoev.ru/en/