Nginx worker processors D state and high I/O utilization !!

[Sergey Kandaurov]

On Jul 16, 2013, at 8:59 AM, shahzaib shahzaib <shahzaib.cb at gmail.com> wrote:
> Hello,
> 
>       We're using nginx-1.2.8 to serve large static files for video streaming. However all nginx worker_processes are in "D" state and HDD I/O utilization is 99%.
> […]
> [root at DNTX010 ~]# iostat -x -d 3
> Linux 2.6.32-358.6.2.el6.x86_64 (DNTX010.local)         07/16/2013      _x86_64_        (8 CPU)
> 
> Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
> sda              30.28   177.37  260.32    2.96 38169.26  1442.70   150.46     2.29    8.70   3.52  92.78
> 
> Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
> sda               4.33     0.00  544.00    0.00 34376.00     0.00    63.19    43.83   75.25   1.84 100.00
> 
> Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
> sda               9.00     6.33  547.67    0.67 34637.33    56.00    63.27    48.01   86.20   1.82 100.00
> 
> Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
> sda               0.00     0.67  568.00    2.33 36024.00    29.33    63.21    54.98  101.10   1.75 100.00
> 
> Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
> sda               0.00     4.33  560.33    1.33 35712.00    45.33    63.66    37.20   65.06   1.78 100.00
> 


You are likely hitting the IOPS limit. 550r/s is quite enough to saturate
4xSATA 7200 in RAID10.  There are reads of 64 of something per request
in average. I'd first look at what does something designated as "rsec/s"
exactly mean in linux terms here.

-- 
Sergey Kandaurov
pluknet at nginx.com