All workers in 'D' state using sendfile

[Host DL]

Dear Maxim,

Thanks for your response,
I don't think it will help much since all of my files are larger enough
than 2MB

Regarding AIO problem in linux, do you think using AIO + sendfile together
on FreeBSD will be better in performance in my case?

Respect



============================================


On Mon, Jun 10, 2013 at 3:25 PM, Maxim Dounin <mdounin at mdounin.ru> wrote:

> Hello!
>
> On Sun, Jun 09, 2013 at 07:24:49PM +0430, Host DL wrote:
>
> > Hello Maxim,
> >
> > Thanks for your response, and sorry that I am new to mailing list and my
> > 1st message may was not very clear to you
> >
> > I've already read all posts in this conversation and all tuning options
> has
> > been tested
> >
> > I'm using 8x 2TB SATA ENT in RAID10 level + 64G RAM on my box
> > CentOS 5.9 x64_84 / 2.6.18-348.6.1.el5
> >
> > nginx.conf:
> >
> > worker_priority -10;
> > worker_processes 64;
> > worker_rlimit_nofile 20000;
> >
> > events {
> >     worker_connections  2048;
> >     use epoll;
> >     worker_aio_requests 128;
> > }
> >
> > http {
> >     sendfile     off;
> >     tcp_nopush     on;
> >     tcp_nodelay on;
> >     aio on;
> >     directio    2m;
> >     #directio_alignment 4k;
> >     output_buffers 1 1m;
> >
> >     keepalive_timeout  15;
> >
> >     ......
> > }
> >
> > During the peak time connections will reach up to 14-15K in total and
> more
> > than 1Gbit/s outgoing throughput
> > Please note that the server was stable with about 12K connections in the
> > peak time and about 1-1.1Gbit/s throughput but after adding another VH
> with
> > about 2-3K connections it seems that server is unable to handle the
> request
> > properly at the peak time
> > Its expected the throughput to exceed the previous ~1.1Gbit/s rate but it
> > doesn't, Even it doesn't reach to 1Gbit/s while the connections are now
> > getting more and bigger
> >
> > During every peak time the LA will each to the number of nginx workers (
> 64
> > for my current config ) and will stay at the same rate to the end of peak
> > time, all processes are in D state and the interesting thing, memory is
> not
> > being used fully and it may push about 30-40G with about 20-30% I/O wait
>
> Main problem with AIO on Linux is that it requires directio to
> actually work asynchronously.  I would assume you've just reached
> a critical number of synchronous requests to disks as due to
> "directio 2m" in your config (and that's why you see all workers
> in D state).  Try tuning directio to a lower value to see if it
> helps.
>
> Note well: memory is not used for filesystem cache with directio,
> so there is no surprise it's not being used fully.
>
> --
> Maxim Dounin
> http://nginx.org/en/donation.html
>
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20130610/6a870383/attachment.html>