<div dir="ltr">Hi,<div><br></div><div> It looks like we made the false calculation based on entertaining concurrent connections per seconds and worker_connections limit was set to be very low. I've increased this limit to 16000 and issue looks to be fixed. Here's the mechanism i used to calculate concurrent connections/sec:<br><br> worker_processes * worker_connections / keepalive_timeout == concurrent connections per second<br><br>Concurrent connections on our server is around 15K . Based on this i used the following values :<br><br> 48 * 16000 / 15 == 51200/sec<br><br>Can somebody point to me if the calculation method is false ?<br><br>Regards.</div><div>Shahzaib</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, May 3, 2015 at 3:11 AM, shahzaib shahzaib <span dir="ltr"><<a href="mailto:shahzaib.cb@gmail.com" target="_blank">shahzaib.cb@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Experts,<div><br></div><div> Could you please do me a favor in order to solve this problem ?<br><br>Regards.</div><span class="HOEnZb"><font color="#888888"><div>Shahzaib</div></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Sat, May 2, 2015 at 3:32 PM, shahzaib shahzaib <span dir="ltr"><<a href="mailto:shahzaib.cb@gmail.com" target="_blank">shahzaib.cb@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div> We've been running nginx-1.8 instance on one of our media server to serve big static .mp4 files as well as small files such as .jpeg. Nginx is serving well under 13K connections/sec with 800Mbps outgoing network load but whenever requests exceed 15K connections, nginx gets halt and 'D' status goes all over around the nginx workers, as well as network load drops down to 400Mbps due to which video streaming gets stuck and after 5-10 minutes load starts dropping and nginx starts stabilizing again as well as network load gets back to 800Mbps. We've been encountering this fluctuating situation on each 15minutes gap (Probably).<br><br> We know that 'D' status is most likely due to high Disk I/O and to ensure that the disk i/o could be the problem under 15K connections, we had enabled apache on port 8080 for testing same video stream during high load and buffered on apache, well the stream was fluctuating a bit but there was no stuck for around 5-10 minutes. In the meantime the same video was worst on nginx and stucked for 5minutes during buffer. <br><br>We suspecting this to be related to something else than Disk I/O, reason is the same video under high load buffers better on apache(on port 8080). Also if it is related to high disk I/O, there must be no possibility that video should should stuck for 5-10 minutes.<br><br>It looks to us that nginx gets halt when concurrent connections exceed 15K. We also tried optimizing backlog directive which slightly improved the performance but there must be something more related to nginx optimization which we must be missing. I have linked nginx.conf file, sysctl and vhost file to get better understanding of our tweaks.<br><br><div>user nginx;</div><div>worker_processes 48;</div><div>worker_rlimit_nofile 600000; #2 filehandlers for each connection</div><div>#error_log logs/error.log;</div><div>#error_log logs/error.log notice;</div><div>error_log /var/log/nginx/error.log error;</div><div>#error_log /dev/null;</div><div>#pid logs/nginx.pid;</div><div><br></div><div><br></div><div>events {</div><div> worker_connections 2048;</div><div> use epoll;</div><div># use kqueue;</div><div>}</div><div>http {</div><div> include mime.types;</div><div> default_type application/octet-stream;</div><div> # client_max_body_size 800M;</div><div> client_body_buffer_size 128K;</div><div> output_buffers 1 512k;</div><div> sendfile_max_chunk 128k;</div><div> client_header_buffer_size 256k;</div><div> large_client_header_buffers 4 256k;</div><div># fastcgi_buffers 512 8k;</div><div># proxy_buffers 512 8k;</div><div># fastcgi_read_timeout 300s;</div><div> server_tokens off; #Conceals nginx version</div><div> access_log off;</div><div># access_log /var/log/nginx/access.log;</div><div> sendfile off;</div><div># sendfile ;</div><div> tcp_nodelay on;</div><div> aio on;</div><div> directio 512;</div><div># tcp_nopush on;</div><div> client_header_timeout 120s;</div><div> client_body_timeout 120s;</div><div> send_timeout 120s;</div><div> keepalive_timeout 15;</div><div> gzip on;</div><div> gzip_vary on;</div><div> gzip_disable "MSIE [1-6]\.";</div><div> gzip_proxied any;</div><div> gzip_http_version 1.0;</div><div> gzip_min_length 1280;</div><div> gzip_comp_level 6;</div><div> gzip_buffers 16 8k;</div><div> gzip_types text/plain text/xml text/css application/x-javascript image/png image/x-icon image/gif image/jpeg image/jpg application/xml application/xml+rss text/javascr ipt application/atom+xml;</div><div> include /usr/local/nginx/conf/vhosts/*.conf;</div><div># open_file_cache max=2000 inactive=20s;</div><div># open_file_cache_valid 60s;</div><div># open_file_cache_min_uses 5;</div><div># open_file_cache_errors off;</div><div><br></div><div>}<br><br>sysctl.conf main config :<br><br><div>fs.file-max = 700000</div><div>net.core.wmem_max=6291456</div><div>net.core.rmem_max=6291456</div><div>net.ipv4.tcp_rmem= 10240 87380 6291456</div><div>net.ipv4.tcp_wmem= 10240 87380 6291456</div><div>net.ipv4.tcp_window_scaling = 1</div><div>net.ipv4.tcp_timestamps = 1</div><div>net.ipv4.tcp_sack = 1</div><div>net.ipv4.tcp_no_metrics_save = 1</div><div>net.core.netdev_max_backlog = 10000</div><div><br></div><div>net.ipv6.conf.all.disable_ipv6 = 1</div><div>net.ipv6.conf.default.disable_ipv6 = 1</div><div>net.ipv6.conf.lo.disable_ipv6 = 1</div><div>net.ipv6.conf.eth0.disable_ipv6 = 1</div><div>net.ipv6.conf.eth1.disable_ipv6 = 1</div><div>net.ipv6.conf.ppp0.disable_ipv6 = 1</div><div>net.ipv6.conf.tun0.disable_ipv6 = 1</div><div>vm.dirty_background_ratio = 50</div><div>vm.dirty_ratio = 80</div><div>net.ipv4.tcp_fin_timeout = 30</div><div>net.ipv4.ip_local_port_range=1024 65000</div><div>net.ipv4.tcp_tw_reuse = 1</div><div>net.netfilter.nf_conntrack_tcp_timeout_established = 54000</div><div>net.ipv4.netfilter.ip_conntrack_generic_timeout = 120</div><div>net.ipv4.tcp_syn_retries=2</div><div>net.ipv4.tcp_synack_retries=2</div><div>net.ipv4.netfilter.ip_conntrack_max = 90536</div><div>net.core.somaxconn = 10000</div></div></div><div><br>Vhost :<br><br><div>server {</div><div> listen 80 backlog=10000;</div><div> server_name <a href="http://archive3.domain.com" target="_blank">archive3.domain.com</a> <a href="http://archive3.domain.com" target="_blank">archive3.domain.com</a> <a href="http://www.archive3.domain.com" target="_blank">www.archive3.domain.com</a> <a href="http://www.archive3.domain.com" target="_blank">www.archive3.domain.com</a>;</div><div> access_log off;</div><div> location / {</div><div> root /content/archive;</div><div> index index.html index.htm index.php;</div><div> autoindex off;</div><div>}</div><div><br></div><div>location /files/thumbs/ {</div><div> root /data/nginx/archive;</div><div> add_header X-Cache SSD;</div><div> expires max;</div><div>}</div><div><br></div><div>location ~ \.(flv)$ {</div><div> flv;</div><div> root /content/archive;</div><div># aio on;</div><div># directio 512;</div><div># output_buffers 1 2m;</div><div> expires 7d;</div><div> valid_referers none blocked <a href="http://domain.com" target="_blank">domain.com</a> *.<a href="http://domain.com" target="_blank">domain.com</a> *.<a href="http://facebook.com" target="_blank">facebook.com</a> *.<a href="http://domain.com" target="_blank">domain.com</a> *.<a href="http://twitter.com" target="_blank">twitter.com</a> *.<a href="http://domain.com" target="_blank">domain.com</a> *.<a href="http://gear3rd.net" target="_blank">gear3rd.net</a> <a href="http://domain.com" target="_blank">domain.com</a> *.<a href="http://domain.com" target="_blank">domain.com</a> <a href="http://tunemedia.tv" target="_blank">tunemedia.tv</a> <a href="http://www.tunemedia.tv" target="_blank">www.tunemedia.tv</a> <a href="http://embed.tunemedia.tv" target="_blank">embed.tunemedia.tv</a>;</div><div> if ($invalid_referer) {</div><div> return 403;</div><div> }</div><div> }</div><div><br></div><div><br></div><div>location ~ \.(mp4)$ {</div><div> mp4;</div><div> mp4_buffer_size 4M;</div><div> mp4_max_buffer_size 10M;</div><div> expires 7d;</div><div> root /content/archive;</div><div> valid_referers none blocked <a href="http://domain.com" target="_blank">domain.com</a> *.<a href="http://domain.com" target="_blank">domain.com</a> *.<a href="http://facebook.com" target="_blank">facebook.com</a> *.<a href="http://domain.com" target="_blank">domain.com</a> *.<a href="http://twitter.com" target="_blank">twitter.com</a> *.<a href="http://domain.com" target="_blank">domain.com</a> *.<a href="http://gear3rd.net" target="_blank">gear3rd.net</a> <a href="http://domain.com" target="_blank">domain.com</a> *.<a href="http://domain.com" target="_blank">domain.com</a> <a href="http://tunemedia.tv" target="_blank">tunemedia.tv</a> <a href="http://www.tunemedia.tv" target="_blank">www.tunemedia.tv</a> <a href="http://embed.tunemedia.tv" target="_blank">embed.tunemedia.tv</a>;</div><div> if ($invalid_referer) {</div><div> return 403;</div><div> }</div><div> }</div><div><br></div><div> # pass the PHP scripts to FastCGI server listening on <a href="http://127.0.0.1:9000" target="_blank">127.0.0.1:9000</a></div><div> location ~ \.php$ {</div><div> root /content/archive;</div><div> fastcgi_pass <a href="http://127.0.0.1:9000" target="_blank">127.0.0.1:9000</a>;</div><div> fastcgi_index index.php;</div><div> fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;</div><div> include fastcgi_params;</div><div> fastcgi_read_timeout 10000;</div><div> }</div><div><br></div><div> location ~ /\.ht {</div><div> deny all;</div><div> }</div><div><br></div><div><br></div><div>location ~ ^/(status|ping)$ {</div><div> access_log off;</div><div> allow 127.0.0.1;</div><div> </div><div> deny all;</div><div> fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;</div><div> include fastcgi_params;</div><div> fastcgi_pass <a href="http://127.0.0.1:9000" target="_blank">127.0.0.1:9000</a>;</div><div>}</div><div>}<br><br>Server Specs :<br><br>L5630 (8cores, 16threads)<br>RAM 64GB<br>12 x 3TB @ SATA Hardware Raid-6<br><br>Here's the screenshot of server load during 15K connections:<br><br><a href="http://prntscr.com/70l68q" target="_blank">http://prntscr.com/70l68q</a><br><br>Regards.</div></div><span><font color="#888888"><div>Shahzaib</div></font></span></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>