Nginx worker processors D state and high I/O utilization !!

[Steve Holdoway]

Your disk maxes out at 20MB/sec read?

On Tue, 2013-07-16 at 09:59 +0500, shahzaib shahzaib wrote:
> Hello,
> 
> 
>       We're using nginx-1.2.8 to serve large static files for video
> streaming. However all nginx worker_processes are in "D" state and HDD
> I/O utilization is 99%.
> 
> [root at DNTX010 ~]# ps aux |grep nginx
> root      3046  0.0  0.0  20272   688 ?        Ss   20:39   0:00
> nginx: master process nginx
> nginx     3047  3.2  0.9  94480 74808 ?        D    20:39   0:03
> nginx: worker process
> nginx     3048  1.4  0.3  52104 31388 ?        D    20:39   0:01
> nginx: worker process
> nginx     3049  0.2  0.1  33156 12156 ?        S    20:39   0:00
> nginx: worker process
> nginx     3050  0.1  0.1  29968  8844 ?        D    20:39   0:00
> nginx: worker process
> nginx     3051  0.2  0.1  30332 10076 ?        D    20:39   0:00
> nginx: worker process
> nginx     3052  2.7  0.8  91788 69504 ?        D    20:39   0:02
> nginx: worker process
> nginx     3053  0.3  0.0  25632  5384 ?        D    20:39   0:00
> nginx: worker process
> nginx     3054  0.2  0.1  36032 15852 ?        D    20:39   0:00
> nginx: worker process
> nginx     3055  0.4  0.2  37592 17396 ?        D    20:39   0:00
> nginx: worker process
> nginx     3056  0.2  0.1  32580 11028 ?        S    20:39   0:00
> nginx: worker process
> nginx     3057  0.3  0.2  39288 19116 ?        D    20:39   0:00
> nginx: worker process
> nginx     3058  0.3  0.2  41764 19744 ?        D    20:39   0:00
> nginx: worker process
> nginx     3059  0.3  0.1  31124 10480 ?        D    20:39   0:00
> nginx: worker process
> nginx     3060  1.0  0.3  52736 31776 ?        D    20:39   0:01
> nginx: worker process
> nginx     3061  1.1  0.3  51920 29956 ?        D    20:39   0:01
> nginx: worker process
> nginx     3062  1.6  0.4  58808 35548 ?        D    20:39   0:01
> nginx: worker process
> 
> 
> [root at DNTX010 ~]# iostat -x -d 3
> Linux 2.6.32-358.6.2.el6.x86_64 (DNTX010.local)         07/16/2013
> _x86_64_        (8 CPU)
> 
> Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s
> avgrq-sz avgqu-sz   await  svctm  %util
> sda              30.28   177.37  260.32    2.96 38169.26  1442.70
> 150.46     2.29    8.70   3.52  92.78
> 
> Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s
> avgrq-sz avgqu-sz   await  svctm  %util
> sda               4.33     0.00  544.00    0.00 34376.00     0.00
> 63.19    43.83   75.25   1.84 100.00
> 
> Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s
> avgrq-sz avgqu-sz   await  svctm  %util
> sda               9.00     6.33  547.67    0.67 34637.33    56.00
> 63.27    48.01   86.20   1.82 100.00
> 
> Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s
> avgrq-sz avgqu-sz   await  svctm  %util
> sda               0.00     0.67  568.00    2.33 36024.00    29.33
> 63.21    54.98  101.10   1.75 100.00
> 
> Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s
> avgrq-sz avgqu-sz   await  svctm  %util
> sda               0.00     4.33  560.33    1.33 35712.00    45.33
> 63.66    37.20   65.06   1.78 100.00
> 
> 
> 
> Nginx.conf :
> 
> http {
>     include       mime.types;
>     default_type  application/octet-stream;
>     client_body_buffer_size 128K;
>     sendfile_max_chunk 128k;
>     client_max_body_size 800m;
>     client_header_buffer_size 256k;
>     large_client_header_buffers 4 256k;
>     output_buffers 1 512k;
>     server_tokens off; #Conceals nginx version
>     #access_log  logs/access.log  main;
>     access_log off;
>     error_log warn;
>     sendfile        on;
> 
> #    aio on;
> 
>  #   directio 512k;
> 
>     ignore_invalid_headers on;
>     client_header_timeout  3m;
>     client_body_timeout 3m;
>     send_timeout     3m;
>     keepalive_timeout  0;
>     reset_timedout_connection on;
> }
> 
> 
> We've also tried enabling aio directive but nothing changed. Help will
> be highly appreciated.
> 
> 
> Thanks
> Shahzaib
> 
> 
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx

-- 
Steve Holdoway BSc(Hons) MNZCS 
http://www.greengecko.co.nz
Linkedin: http://www.linkedin.com/in/steveholdoway
Skype: sholdowa