Hi Ryan,<br><br>If I comment out use epoll;, it still seems to use epoll according to strace. These are the compile flags I use:<br><br>./configure --prefix=/usr/local/nginx-1.2.0 --with-pcre --add-module=/usr/local/src/nginx_upload_module-2.2.0 --with-http_stub_status_module --with-file-aio --without-http_proxy_module --without-http_memcached_module --without-http_gzip_module --without-http_ssi_module --without-http_userid_module --without-http_autoindex_module --without-http_geo_module --without-http_map_module --without-http_empty_gif_module --without-http_browser_module --without-http_upstream_ip_hash_module --without-http_charset_module<br>
<br>This is what I'm running with now:<br><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace">events {</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace"> worker_connections 51200;</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace"># use epoll;</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace"># multi_accept on;</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">}</span><br style="font-family:courier new,monospace"><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace">http {</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace"> include mime.types;</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace"> default_type application/octet-stream;</span><br style="font-family:courier new,monospace">
<br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace"> server_tokens off;</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace"> sendfile off;</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace"> tcp_nopush on;</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace"> tcp_nodelay on;</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace"> keepalive_timeout 10;</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace"> </span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace"> aio on;</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace"> directio 4k;</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace"> output_buffers 1 512k;</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace"> max_ranges 5;</span><br>
<span style="font-family:courier new,monospace"> ...</span><br clear="all"><br><br>Cheers,<br><br>Drew<br>
<br><br><div class="gmail_quote">On Tue, May 29, 2012 at 12:31 AM, Ryan Brown <span dir="ltr"><<a href="mailto:mp3geek@gmail.com" target="_blank">mp3geek@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
If you comment/remove "use epoll;" does that help?<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
On Mon, May 28, 2012 at 10:06 PM, 姚伟斌 <<a href="mailto:nbubingo@gmail.com">nbubingo@gmail.com</a>> wrote:<br>
> Hi Maxim,<br>
><br>
> Is there any planning to develop the thread version?<br>
><br>
> 2012/5/12 Maxim Dounin <<a href="mailto:mdounin@mdounin.ru">mdounin@mdounin.ru</a>>:<br>
>> Hello!<br>
>><br>
>> On Sat, May 12, 2012 at 08:28:14PM +1000, Drew Wareham wrote:<br>
>><br>
>>> Hello,<br>
>>><br>
>>> I have tried to summarize this as much as possible but it's still a lot of<br>
>>> text. I apologize but wanted to make sure that I provide enough<br>
>>> information to explain the issue properly.<br>
>>><br>
>>> I'm hoping that somebody that uses nginx as a high traffic/concurrency<br>
>>> download server will be able to shed some light on this issue. I've tried<br>
>>> as many things as I can think of and everything keeps pointing to it being<br>
>>> an issue with nginx, not the server - but I am of course more than willing<br>
>>> to try any suggestions provided.<br>
>>><br>
>>> *Background:*<br>
>>> Approx. 1,500 - 5,000 concurrent connections (peak / off-peak),<br>
>>> Files vary in size from 5MB to 2GB,<br>
>>> All downloads; only very small dynamic content scripts run on these servers<br>
>>> and none take more than 1-3 seconds,<br>
>>> File are hosted on direct-attached AoE storage with a dedicated 10GE link,<br>
>>> Server is running nginx-1.0.11, php-fpm 5.3 and CentOS 5.8x64<br>
>>> (2.6.18-308.4.1.el5.centos.plus).<br>
>>> Specs are: Dual Xeon E5649 (6 Core), 32GB RAM, 300GB 10k SAS HDD, AoE DAS<br>
>>> over 10GE<br>
>>> Download speeds are restricted by the PHP handoff using X-Accel-Redirect,<br>
>>> but obviously not when I'm testing ;)<br>
>>><br>
>>> *Issue:*<br>
>>> After running for a short, but random period of time (5min ~ 90min) all<br>
>>> nginx workers will eventually end up in a 'D' state according to ps/top.<br>
>>> This causes all downloads to run extremely slowly (~25kb/s) but it doesn't<br>
>>> seem to be caused by I/O because an scp of the same file will complete at<br>
>>> the expected speed of ~750MB+/s.<br>
>>><br>
>>> I usually run with worker_processes set to 13, but I've had to raise this<br>
>>> to 50 to prevent the issue. This works short term, but I'm guessing<br>
>>> eventually I will need to restart nginx to fix it.<br>
>>><br>
>>> *Config:*<br>
>>> I'm using sendfile with epoll, and using the following events / http<br>
>>> settings (I've removed the location block with the fastcgi handler, etc):<br>
>><br>
>> With rotational disks you have to optimize iops to minimize seeks.<br>
>> This includes:<br>
>><br>
>> 1. Switch off sendfile, it works bad on such workloads under linux<br>
>> due to no ability to control readahead (and hence blocks read from<br>
>> disk).<br>
>><br>
>> 2. Use large output buffers, something like<br>
>><br>
>> output_buffers 1 512k<br>
>><br>
>> would be a good starting point.<br>
>><br>
>> 3. Try using aio to ensure better disk concurrency (and note under<br>
>> linux it needs directio as well), i.e. something like this<br>
>><br>
>> aio on;<br>
>> directio 512;<br>
>><br>
>> (this will require newer kernel though, but using 2.6.18 nowadays<br>
>> looks like bad idea, at least if you need speed)<br>
>><br>
>> 4. Try tuning io scheduler, there have been reports that deadline<br>
>> might be better for such workloads.<br>
>><br>
>> More details can be found here:<br>
>><br>
>> <a href="http://nginx.org/r/output_buffers" target="_blank">http://nginx.org/r/output_buffers</a><br>
>> <a href="http://nginx.org/r/aio" target="_blank">http://nginx.org/r/aio</a><br>
>> <a href="http://nginx.org/r/directio" target="_blank">http://nginx.org/r/directio</a><br>
>><br>
>> Maxim Dounin<br>
>><br>
>> _______________________________________________<br>
>> nginx mailing list<br>
>> <a href="mailto:nginx@nginx.org">nginx@nginx.org</a><br>
>> <a href="http://mailman.nginx.org/mailman/listinfo/nginx" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a><br>
><br>
> _______________________________________________<br>
> nginx mailing list<br>
> <a href="mailto:nginx@nginx.org">nginx@nginx.org</a><br>
> <a href="http://mailman.nginx.org/mailman/listinfo/nginx" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a><br>
<br>
_______________________________________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx" target="_blank">http://mailman.nginx.org/mailman/listinfo/nginx</a></div></div></blockquote></div><br>