High number connections-writing stuck

Sat Dec 19 17:08:19 UTC 2015

>From what you provided, I do not see anything unexpected...

Enabling threads usage with thread_pool and aio threads enables
asynchronous file reading/writing.
The connections_writing stat indicates... the number of connections to
which the server is writing a response.

Since you are working asynchronously, if the request takes much less time
to process than the response to be sent, it is normal this numbers
increases, until reaching a balance point.

You saw this number increasing when you reloaded the configuration -> shows
the new configuration has been applied immediately
You saw this number drop on restart -> restarting kills the current
processes and the attached connections
Nothing to see on CPU & RAM -> they are probably not the bottleneck, and
one nginx strength is to have a minimal fingerprint there, compared to
other Web server alternatives. Have a look at I/O wait to check if the
worker processes might be having a hard time getting content they seek from
the disk. Less often, it might also be a congestion on backend
communication if content is being read from some.

To me, it is just a matter of raton time to process request/time to serve
answer. Multithreading means a worker might be answering several
connections at the same time.
If I am right about nginx' design, events might be processed in any order,
but each worker still only does one thing at a time and onky switch task
when the current one is finished/waiting on something else. Multithreading
changes the whole behavior.

Hope to help,
---
*B. R.*

On Fri, Dec 18, 2015 at 6:55 PM, Marcelo MD <lists.md at gmail.com> wrote:

> Hi,
>
> Recently we added a 'thread_pool' directive to our main configuration. A
> few hours later we saw a huge increase in the connections_writing stat as
> reported by stub_status module. This number reached +- 3800 and is stuck
> there since. The server in question is operating normally, but this is very
> strange.
>
> Any hints on what this could be?
>
>
> Some info:
>
> - Here is a graph of the stats reported, for a server with thread_pool and
> another without: http://imgur.com/a/lF2EL
>
> - I don`t have older data anymore, but the jump from <100 to +- 3800
> connections_writing happened in two sharp jumps. The first one following a
> reload;
>
> - The machines' hardware and software are identical except for the
> thread_pool directive in their nginx.conf. They live in two different data
> centers;
>
> - Both machines are performing normally. Nothing unusual in CPU or RAM
> usage. Nginx performance is about the same.
>
> - Reloading Nginx with 'nginx -s reload' does nothing. Restarting the
> process brings connections_writing down.
>
>
> Debug stuff:
>
> mallmann# uname -a
> Linux xxx 3.8.13-98.5.2.el6uek.x86_64 #2 SMP Tue Nov 3 18:32:04 PST 2015
> x86_64 x86_64 x86_64 GNU/Linux
>
> mallmann# nginx -V
> nginx version: nginx/1.8.0
> built by gcc 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC)
> built with OpenSSL 1.0.1e-fips 11 Feb 2013
> TLS SNI support enabled
> configure arguments: --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx
> --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log
> --http-log-path=/var/log/nginx/access.log
> --http-client-body-temp-path=/var/lib/nginx/tmp/client_body
> --http-proxy-temp-path=/var/lib/nginx/tmp/proxy
> --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi
> --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi
> --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/var/run/nginx.pid
> --lock-path=/var/lock/subsys/nginx --user=nginx --group=nginx --with-ipv6
> --with-http_ssl_module --with-http_realip_module
> --with-http_addition_module --with-http_xslt_module
> --with-http_image_filter_module --with-http_geoip_module
> --with-http_sub_module --with-http_flv_module --with-http_mp4_module
> --with-http_gunzip_module --with-http_gzip_static_module
> --with-http_random_index_module --with-http_secure_link_module
> --with-http_degradation_module --with-http_stub_status_module
> --with-http_perl_module --with-mail --with-mail_ssl_module --with-pcre
> --with-google_perftools_module
> --add-module=/builddir/build/BUILD/nginx-1.8.0/headers-more-nginx-module-0.25
> --add-module=/builddir/build/BUILD/nginx-1.8.0/ngx_http_bytes_filter_module
> --add-module=/builddir/build/BUILD/nginx-1.8.0/echo-nginx-module-0.55
> --with-threads --with-debug --with-cc-opt='-O2 -g -pipe -Wall
> -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
> --param=ssp-buffer-size=4 -m64 -mtune=generic' --with-ld-opt=' -Wl,-E'
>
>
> Affected server:
>
> mallmann# lsof -n -u nginx | awk '{print $5}' | sort | uniq -c | sort -nr
>    4172 REG
>    2140 IPv4
>     100 unix
>      30 CHR
>      20 DIR
>      20 0000
>       3 sock
>       1 TYPE
> mallmann# curl http://127.0.0.1/status
> Active connections: 5924
> server accepts handled requests
>  5864099 5864099 15527178
> Reading: 0 Writing: 3883 Waiting: 2040
>
>
> Normal server:
>
> mallmann# lsof -n -u nginx | awk '{print $5}' | sort | uniq -c | sort -nr
>    4454 REG
>    1967 IPv4
>     100 unix
>      30 CHR
>      20 DIR
>      20 0000
>       1 unknown
>       1 TYPE
>       1 sock
> mallmann# curl http://127.0.0.1/status
> Active connections: 2096
> server accepts handled requests
>  1136132 1136132 3464904
> Reading: 0 Writing: 107 Waiting: 1989
>
> --
> Marcelo Mallmann Dias
>
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20151219/2dcafb72/attachment.html>