High number connections-writing stuck

Marcelo MD lists.md at gmail.com
Fri Dec 18 17:55:47 UTC 2015


Recently we added a 'thread_pool' directive to our main configuration. A
few hours later we saw a huge increase in the connections_writing stat as
reported by stub_status module. This number reached +- 3800 and is stuck
there since. The server in question is operating normally, but this is very

Any hints on what this could be?

Some info:

- Here is a graph of the stats reported, for a server with thread_pool and
another without: http://imgur.com/a/lF2EL

- I don`t have older data anymore, but the jump from <100 to +- 3800
connections_writing happened in two sharp jumps. The first one following a

- The machines' hardware and software are identical except for the
thread_pool directive in their nginx.conf. They live in two different data

- Both machines are performing normally. Nothing unusual in CPU or RAM
usage. Nginx performance is about the same.

- Reloading Nginx with 'nginx -s reload' does nothing. Restarting the
process brings connections_writing down.

Debug stuff:

mallmann# uname -a
Linux xxx 3.8.13-98.5.2.el6uek.x86_64 #2 SMP Tue Nov 3 18:32:04 PST 2015
x86_64 x86_64 x86_64 GNU/Linux

mallmann# nginx -V
nginx version: nginx/1.8.0
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC)
built with OpenSSL 1.0.1e-fips 11 Feb 2013
TLS SNI support enabled
configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx
--conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log
--http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/var/run/nginx.pid
--lock-path=/var/lock/subsys/nginx --user=nginx --group=nginx --with-ipv6
--with-http_ssl_module --with-http_realip_module
--with-http_addition_module --with-http_xslt_module
--with-http_image_filter_module --with-http_geoip_module
--with-http_sub_module --with-http_flv_module --with-http_mp4_module
--with-http_gunzip_module --with-http_gzip_static_module
--with-http_random_index_module --with-http_secure_link_module
--with-http_degradation_module --with-http_stub_status_module
--with-http_perl_module --with-mail --with-mail_ssl_module --with-pcre
--with-threads --with-debug --with-cc-opt='-O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m64 -mtune=generic' --with-ld-opt=' -Wl,-E'

Affected server:

mallmann# lsof -n -u nginx | awk '{print $5}' | sort | uniq -c | sort -nr
   4172 REG
   2140 IPv4
    100 unix
     30 CHR
     20 DIR
     20 0000
      3 sock
      1 TYPE
mallmann# curl
Active connections: 5924
server accepts handled requests
 5864099 5864099 15527178
Reading: 0 Writing: 3883 Waiting: 2040

Normal server:

mallmann# lsof -n -u nginx | awk '{print $5}' | sort | uniq -c | sort -nr
   4454 REG
   1967 IPv4
    100 unix
     30 CHR
     20 DIR
     20 0000
      1 unknown
      1 TYPE
      1 sock
mallmann# curl
Active connections: 2096
server accepts handled requests
 1136132 1136132 3464904
Reading: 0 Writing: 107 Waiting: 1989

Marcelo Mallmann Dias
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20151218/ac0e5f2b/attachment.html>

More information about the nginx mailing list