<div dir="ltr"><div class="gmail_default"><div class="gmail_default"><font face="tahoma, sans-serif">Hello Maxim,</font></div><div class="gmail_default"><font face="tahoma, sans-serif"><br></font></div><div class="gmail_default">
<font face="tahoma, sans-serif">Thanks for your response, and sorry that I am new to mailing list and my 1st message may was not very clear to you</font></div><div class="gmail_default"><font face="tahoma, sans-serif"><br>
</font></div><div class="gmail_default"><font face="tahoma, sans-serif">I've already read all posts in this conversation and all tuning options has been tested</font></div><div class="gmail_default"><font face="tahoma, sans-serif"><br>
</font></div><div class="gmail_default"><font face="tahoma, sans-serif">I'm using 8x 2TB SATA ENT in RAID10 level + 64G RAM on my box</font></div><div class="gmail_default"><font face="tahoma, sans-serif">CentOS 5.9 x64_84 / 2.6.18-348.6.1.el5</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif"><br></font></div><div class="gmail_default"><font face="tahoma, sans-serif">nginx.conf:</font></div><div class="gmail_default"><font face="tahoma, sans-serif"><br>
</font></div><div class="gmail_default"><font face="tahoma, sans-serif">worker_priority -10;</font></div><div class="gmail_default"><font face="tahoma, sans-serif">worker_processes 64;</font></div><div class="gmail_default">
<font face="tahoma, sans-serif">worker_rlimit_nofile 20000;</font></div><div class="gmail_default"><font face="tahoma, sans-serif"><br></font></div><div class="gmail_default"><font face="tahoma, sans-serif">events {</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif"> worker_connections 2048;</font></div><div class="gmail_default"><font face="tahoma, sans-serif"> use epoll;</font></div><div class="gmail_default"><font face="tahoma, sans-serif"> worker_aio_requests 128;</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif">}</font></div><div class="gmail_default"><font face="tahoma, sans-serif"><br></font></div><div class="gmail_default"><font face="tahoma, sans-serif">http {</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif"> sendfile off;</font></div><div class="gmail_default"><font face="tahoma, sans-serif"> tcp_nopush on;</font></div><div class="gmail_default"><font face="tahoma, sans-serif"> tcp_nodelay on;</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif"> aio on;</font></div><div class="gmail_default"><font face="tahoma, sans-serif"> directio 2m;</font></div><div class="gmail_default"><font face="tahoma, sans-serif"> #directio_alignment 4k;</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif"> output_buffers 1 1m;</font></div><div class="gmail_default"><font face="tahoma, sans-serif"><br></font></div><div class="gmail_default"><font face="tahoma, sans-serif"> keepalive_timeout 15;</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif"><br></font></div><div class="gmail_default"><font face="tahoma, sans-serif"> ......</font></div><div class="gmail_default"><font face="tahoma, sans-serif">}</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif"><br></font></div><div class="gmail_default"><font face="tahoma, sans-serif">During the peak time connections will reach up to 14-15K in total and more than 1Gbit/s outgoing throughput</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif">Please note that the server was stable with about 12K connections in the peak time and about 1-1.1Gbit/s throughput but after adding another VH with about 2-3K connections it seems that server is unable to handle the request properly at the peak time</font></div>
<div class="gmail_default" style><font face="tahoma, sans-serif">Its expected the throughput to exceed the previous ~1.1Gbit/s rate but it doesn't, Even it doesn't reach to 1Gbit/s while the connections are now getting more and bigger</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif"><br></font></div><div class="gmail_default"><font face="tahoma, sans-serif">During every peak time the LA will each to the number of nginx workers ( 64 for my current config ) and will stay at the same rate to the end of peak time, all processes are in D state and the interesting thing, memory is not being used fully and it may push about 30-40G with about 20-30% I/O wait</font></div>
<div class="gmail_default" style><font face="tahoma, sans-serif"><br></font></div><div class="gmail_default"><font face="tahoma, sans-serif">Connections are not being processed fast and it will stay in connecting and request sent/wait for a few seconds and then data transfer will start at a very slow rate</font></div>
<div class="gmail_default"><font face="tahoma, sans-serif"><br></font></div><div class="gmail_default" style><font face="tahoma, sans-serif">I've already tried both sendfile and AIO, and AIO seems to handle the connections better</font></div>
<div class="gmail_default" style><font face="tahoma, sans-serif">Played with output_buffer and increased both number and size of buffers but no any special effect, even the throughput got lower</font></div><div class="gmail_default" style>
<font face="tahoma, sans-serif"><br></font></div><div class="gmail_default" style><font face="tahoma, sans-serif">The interesting thing is that at the same peak time the read transfer rate is more than 500mbit/s using scp/rsync from the RAID to /dev/null</font></div>
<div class="gmail_default" style><font face="tahoma, sans-serif"><br></font></div><div class="gmail_default" style><font face="tahoma, sans-serif">Sorry for my long post and if my english sux</font></div><div class="gmail_default" style>
<font face="tahoma, sans-serif">Any suggestion would be greatly appreciarted</font></div><div class="gmail_default" style><font face="tahoma, sans-serif"><br></font></div><div class="gmail_default" style><font face="tahoma, sans-serif">Respect,</font></div>
<div class="gmail_default" style><font face="tahoma, sans-serif">Moozi</font></div><div class="gmail_default" style><font face="tahoma, sans-serif"><br></font></div></div><div class="gmail_default" style="font-family:tahoma,sans-serif">
<br></div></div><div class="gmail_extra"><br clear="all"><div><div dir="ltr"><div style="text-align:center"><span style="font-family:tahoma,sans-serif"><br></span></div><div style="text-align:center"><span style="font-family:tahoma,sans-serif">======================</span>======================</div>
<div><div><div><div><div><div><div><div><font face="tahoma, sans-serif"></font><font face="tahoma, sans-serif"></font></div></div></div></div></div></div></div></div></div></div>
<br><br><div class="gmail_quote">On Sun, Jun 9, 2013 at 4:43 PM, Maxim Dounin <span dir="ltr"><<a href="mailto:mdounin@mdounin.ru" target="_blank">mdounin@mdounin.ru</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello!<br>
<div class="im"><br>
On Sun, Jun 09, 2013 at 06:17:10AM +0430, Host DL wrote:<br>
<br>
> I am facing the same exact issue as explained by Drew,<br>
><br>
> is there any working solution to tune nginx for higher throughput?<br>
><br>
> or how to deal with sleeping D state nginx processes ?<br>
<br>
</div>See this reply for basic tuning suggestions:<br>
<br>
<a href="http://mailman.nginx.org/pipermail/nginx/2012-May/033761.html" target="_blank">http://mailman.nginx.org/pipermail/nginx/2012-May/033761.html</a><br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
Maxim Dounin<br>
<a href="http://nginx.org/en/donation.html" target="_blank">http://nginx.org/en/donation.html</a><br>
<br>
_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a><br>
</font></span></blockquote></div><br></div>