All workers in 'D' state using sendfile

[Maxim Dounin]

Hello!

On Sun, Jun 09, 2013 at 07:24:49PM +0430, Host DL wrote:

> Hello Maxim,
> 
> Thanks for your response, and sorry that I am new to mailing list and my
> 1st message may was not very clear to you
> 
> I've already read all posts in this conversation and all tuning options has
> been tested
> 
> I'm using 8x 2TB SATA ENT in RAID10 level + 64G RAM on my box
> CentOS 5.9 x64_84 / 2.6.18-348.6.1.el5
> 
> nginx.conf:
> 
> worker_priority -10;
> worker_processes 64;
> worker_rlimit_nofile 20000;
> 
> events {
>     worker_connections  2048;
>     use epoll;
>     worker_aio_requests 128;
> }
> 
> http {
>     sendfile     off;
>     tcp_nopush     on;
>     tcp_nodelay on;
>     aio on;
>     directio    2m;
>     #directio_alignment 4k;
>     output_buffers 1 1m;
> 
>     keepalive_timeout  15;
> 
>     ......
> }
> 
> During the peak time connections will reach up to 14-15K in total and more
> than 1Gbit/s outgoing throughput
> Please note that the server was stable with about 12K connections in the
> peak time and about 1-1.1Gbit/s throughput but after adding another VH with
> about 2-3K connections it seems that server is unable to handle the request
> properly at the peak time
> Its expected the throughput to exceed the previous ~1.1Gbit/s rate but it
> doesn't, Even it doesn't reach to 1Gbit/s while the connections are now
> getting more and bigger
> 
> During every peak time the LA will each to the number of nginx workers ( 64
> for my current config ) and will stay at the same rate to the end of peak
> time, all processes are in D state and the interesting thing, memory is not
> being used fully and it may push about 30-40G with about 20-30% I/O wait

Main problem with AIO on Linux is that it requires directio to 
actually work asynchronously.  I would assume you've just reached 
a critical number of synchronous requests to disks as due to 
"directio 2m" in your config (and that's why you see all workers 
in D state).  Try tuning directio to a lower value to see if it 
helps.

Note well: memory is not used for filesystem cache with directio, 
so there is no surprise it's not being used fully.

-- 
Maxim Dounin
http://nginx.org/en/donation.html