<div dir="ltr">Doing strace on a nginx child in the shutdown state i get <div><br></div><div>##################</div><div><div> strace -p 23846</div><div>strace: Process 23846 attached</div><div>restart_syscall(<... resuming interrupted futex ...></div><div><br></div><div>) = -1 ETIMEDOUT (Connection timed out)</div><div>futex(0x7a6c0078, FUTEX_WAKE_PRIVATE, 1) = 0</div><div>futex(0x7a6c00c4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 5395, {1504781553, 30288000}, ffffffff</div><div><br></div><div>) = -1 ETIMEDOUT (Connection timed out)</div><div>futex(0x7a6c0078, FUTEX_WAKE_PRIVATE, 1) = 0</div><div>futex(0x7a6c00c4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 5397, {1504781554, 30408000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)</div><div>futex(0x7a6c0078, FUTEX_WAKE_PRIVATE, 1) = 0</div><div>futex(0x7a6c00c4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 5399, {1504781555, 30535000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)</div><div>futex(0x7a6c0078, FUTEX_WAKE_PRIVATE, 1) = 0</div><div>futex(0x7a6c00c4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 5401, {1504781556, 30675000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)</div><div>futex(0x7a6c0078, FUTEX_WAKE_PRIVATE, 1) = 0</div><div>futex(0x7a6c00c4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 5403, {1504781557, 30767000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)</div><div>futex(0x7a6c0078, FUTEX_WAKE_PRIVATE, 1) = 0</div><div>futex(0x7a6c00c4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 5405, {1504781558, 30889000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)</div><div>futex(0x7a6c0078, FUTEX_WAKE_PRIVATE, 1) = 0</div><div>futex(0x7a6c00c4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 5407, {1504781559, 30980000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)</div><div>futex(0x7a6c0078, FUTEX_WAKE_PRIVATE, 1) = 0</div><div>futex(0x7a6c00c4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 5409, {1504781560, 31099000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)</div><div>futex(0x7a6c0078, FUTEX_WAKE_PRIVATE, 1) = 0</div><div>futex(0x7a6c00c4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 5411, {1504781561, 31210000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)</div><div>futex(0x7a6c0078, FUTEX_WAKE_PRIVATE, 1) = 0</div><div>futex(0x7a6c00c4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 5413, {1504781562, 31317000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)</div><div>futex(0x7a6c0078, FUTEX_WAKE_PRIVATE, 1) = 0</div><div>futex(0x7a6c00c4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 5415, {1504781563, 31428000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)</div><div>futex(0x7a6c0078, FUTEX_WAKE_PRIVATE, 1) = 0</div><div>futex(0x7a6c00c4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 5417, {1504781564, 31575000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)</div><div>futex(0x7a6c0078, FUTEX_WAKE_PRIVATE, 1) = 0</div><div>futex(0x7a6c00c4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 5419, {1504781565, 31678000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)</div><div>futex(0x7a6c0078, FUTEX_WAKE_PRIVATE, 1) = 0</div><div>futex(0x7a6c00c4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 5421, {1504781566, 31828000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)</div><div>futex(0x7a6c0078, FUTEX_WAKE_PRIVATE, 1) = 0</div><div>futex(0x7a6c00c4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 5423, {1504781567, 31941000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)</div><div>futex(0x7a6c0078, FUTEX_WAKE_PRIVATE, 1) = 0</div><div>futex(0x7a6c00c4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 5425, {1504781568, 32085000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)</div></div><div>###############################################</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Sep 7, 2017 at 3:59 PM, Lucas Rolff <span dir="ltr"><<a href="mailto:lucas@lucasrolff.com" target="_blank">lucas@lucasrolff.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="white" lang="EN-GB" link="#0563C1" vlink="#954F72">
<div class="m_-2570984022319663013WordSection1">
<p class="MsoNormal">Check if any of the sites you run on the server gets crawled by any crawlers around the time you see an increase – I know that a crawler such as Screaming Frog doesn’t handle servers that are capable of http2 connections and have it activated
for sites that are getting crawled, and will result in connections with a “waiting” state in nginx.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">It might be there’s other tools that behave the same way, but I’d personally look into what kind of traffic/requests happened that increased the waiting state a lot.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Best Regards,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:36.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">nginx <<a href="mailto:nginx-bounces@nginx.org" target="_blank">nginx-bounces@nginx.org</a>> on behalf of Anoop Alias <<a href="mailto:anoopalias01@gmail.com" target="_blank">anoopalias01@gmail.com</a>><br>
<b>Reply-To: </b>"<a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a>" <<a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a>><br>
<b>Date: </b>Thursday, 7 September 2017 at 11.52<br>
<b>To: </b>Nginx <<a href="mailto:nginx@nginx.org" target="_blank">nginx@nginx.org</a>><br>
<b>Subject: </b>Too many connections in waiting state<u></u><u></u></span></p>
</div><div><div class="h5">
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">Hi, <u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">I see sometimes too many waiting connections on nginx .<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">This often gets cleared on a restart , but otherwise pileup<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">###################<u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">Active connections: 4930 <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">server accepts handled requests <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"> 442071 442071 584163 <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">Reading: 2 Writing: 539 Waiting: 4420 <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">#######################<u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">[root@web1 ~]# grep keep /etc/nginx/conf.d/http_<wbr>settings_custom.conf<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">keepalive_timeout 10s;<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">keepalive_requests 200;<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">keepalive_disable msie6 safari;<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">########################<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">[root@web1 ~]# nginx -V<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">nginx version: nginx/1.13.3<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">built by gcc 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">built with LibreSSL 2.5.5<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">TLS SNI support enabled<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/etc/nginx/<wbr>modules --with-pcre=./pcre-8.41 --with-pcre-jit --with-zlib=./zlib-1.2.11 --with-openssl=./libressl-2.5.<wbr>5 --conf-path=/etc/nginx/nginx.<wbr>conf
--error-log-path=/var/log/<wbr>nginx/error_log --http-log-path=/var/log/<wbr>nginx/access_log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.<wbr>lock --http-client-body-temp-path=/<wbr>var/cache/nginx/client_temp --http-proxy-temp-path=/var/<wbr>cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/<wbr>cache/nginx/fastcgi_temp
--http-uwsgi-temp-path=/var/<wbr>cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/<wbr>cache/nginx/scgi_temp --user=nobody --group=nobody --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module
--with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_<wbr>module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_<wbr>module --add-dynamic-module=naxsi-<wbr>http2/naxsi_src --with-file-aio
--with-threads --with-stream --with-stream_ssl_module --with-http_slice_module --with-compat --with-http_v2_module --with-http_geoip_module=<wbr>dynamic --add-dynamic-module=ngx_<wbr>pagespeed-1.12.34.2-stable --add-dynamic-module=/usr/<wbr>local/rvm/gems/ruby-2.4.1/<wbr>gems/passenger-5.1.8/src/<wbr>nginx_module
--add-dynamic-module=ngx_<wbr>brotli --add-dynamic-module=echo-<wbr>nginx-module-0.60 --add-dynamic-module=headers-<wbr>more-nginx-module-0.32 --add-dynamic-module=ngx_http_<wbr>redis-0.3.8 --add-dynamic-module=redis2-<wbr>nginx-module --add-dynamic-module=srcache-<wbr>nginx-module-0.31
--add-dynamic-module=ngx_<wbr>devel_kit-0.3.0 --add-dynamic-module=set-misc-<wbr>nginx-module-0.31 --add-dynamic-module=<wbr>testcookie-nginx-module --add-dynamic-module=<wbr>ModSecurity-nginx --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
--param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic' --with-ld-opt=-Wl,-E<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">#######################<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">What could be causing this? The server is quite capable and this happens only rarely<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><u></u> <u></u></p>
</div>
<p class="MsoNormal" style="margin-left:36.0pt">-- <u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><b>Anoop P Alias</b> <u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div></div></div>
</div>
<br>______________________________<wbr>_________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx" rel="noreferrer" target="_blank">http://mailman.nginx.org/<wbr>mailman/listinfo/nginx</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><b>Anoop P Alias</b> <div><br></div></div></div></div>
</div>