Identifying "Writing" connections in status stub
Peter Booth
peter_booth at me.com
Sun Jul 30 11:30:25 UTC 2017
See below
> On Jul 30, 2017, at 6:12 AM, Vlad K. <nginx-ml at acheronmedia.hr> wrote:
>
> On 2017-07-30 11:26, Peter Booth wrote:
>> I just reread the thread and realize that you answered q2, and that
>> makes the graph even more
>> surprising. You say that it son FreeBSD - does this mean that you
>> don’t have /proc available to you?
>> Is there a procstat or other way to see the equivalent of
>> /proc/<pid>/fd - a list of all open file descriptions for a specific
>> pid?
>
> procfs is technically available but I'm not enabling it. We do have sockstat and fstat though that basically can show the same. I have checked and I don't see any corresponding open connections, that's why I was wondering about how to list what nginx thinks are connections being Written to, so I could find what those connections did last in the access log.
It appears that you have a lot of data that could help in this analysis.
How frequently is the status page being queried? Does every status datapoint get recorded
or is munin showing some rolled up rrd data?
>
> I ran a periodic netstat, sorted the IPs (not removing src ports), and ran a diff from previous run, keeping only those that aren't +/-. Over some period of time, the only connections that stayed open for long were actively using the roundcube webmail which (especially with our keepalive and http2 enabled) runs a request every minute. But the number of those did not correspond to the number reported as Writing. Plus these connections have daily highs and downs. "Writing" in the nginx status does not correlate with the rest.
If you open the status page in a browser do the numbers report match what you see with netstat?
>
>
>> 1. When is the last time that your production nginx was restarted?
>
> In my first post in this thread is a link to the graph (by Munin) where it shows a dip in Writing, that's where I restarted. Reloading doesn't change the number of Writing reported by nginx.
Thats what I thought. I think that the graph looks weird. Over the time interval [18 July, 23 July] the time series
labelled “writing” connections increases almost monotonically from approx 5 to 15. Then the nginx is restarted
and the graph jumps back unto about 12/13 and continues writing again. Do you have a hypothesis that explains
why the graph could jump back to 12/13, rather than spend a few days increasing linearly in the way it did from
the 18th to the 23rd?
How long was nginx down for? If you graph only the “writing” variable for just 23rd July does the length of
time that the # of writing connections is thoughtto be 0 make sense?
I wonder whether what you are seeing could be a side-effect of the server being in a FreeBSD jail?
I haven’t used FreeBSD. My understanding is that FreeBSD jails are more than just chroots, similar to Solaris Zones
or OpenVZ on Linux. Does each jail have separate, independent sysctl settings/ulimits?
Do any of the other nginx sites in other jails exhibit the same behavior?
In FreeBSD jails is there an equivalent of Dom) in a XEN hypervisor? A parent or root OS?
If so, do you see all connections on al jails the you log into it? If wondering if you are hitting some ulimit or
resource shortage on the host as a whole?
>
>
>> 2. Do you have regular restarts?
>
> No, just regular reloads.
>
>
>> 3. Is there an obstacle to restarting at some point?
>
> If you're asking me if I can restart nginx to check something, I can do that.
>
>
>> 4. Is this a single instance or do you have multiple nginx hosts?
>
> Single instance, one master and one worker thread, in a jail, and there's no other jail running nginx on the server.
>
>
>> 5. What 3rd party models are you using?
>
> These are the options/modules ENABLED for the port:
>
> DSO=on: Enable dynamic modules support
> FILE_AIO=on: Enable file aio
> HTTP=on: Enable HTTP module
> HTTPV2=on: Enable HTTP/2 protocol support (SSL req.)
> HTTP_ADDITION=on: Enable http_addition module
> HTTP_AUTH_REQ=on: Enable http_auth_request module
> HTTP_CACHE=on: Enable http_cache module
> HTTP_DAV=on: Enable http_webdav module
> HTTP_FLV=on: Enable http_flv module
> HTTP_GUNZIP_FILTER=on: Enable http_gunzip_filter module
> HTTP_GZIP_STATIC=on: Enable http_gzip_static module
> HTTP_MP4=on: Enable http_mp4 module
> HTTP_RANDOM_INDEX=on: Enable http_random_index module
> HTTP_REALIP=on: Enable http_realip module
> HTTP_REWRITE=on: Enable http_rewrite module
> HTTP_SECURE_LINK=on: Enable http_secure_link module
> HTTP_SLICE=on: Enable http_slice module
> HTTP_SSL=on: Enable http_ssl module
> HTTP_STATUS=on: Enable http_stub_status module
> HTTP_SUB=on: Enable http_sub module
> IPV6=on: Enable IPv6 support
> MAIL=on: Enable IMAP4/POP3/SMTP proxy module
> MAIL_IMAP=on: Enable IMAP4 proxy module
> MAIL_POP3=on: Enable POP3 proxy module
> MAIL_SMTP=on: Enable SMTP proxy module
> MAIL_SSL=on: Enable mail_ssl module
> STREAM=on: Enable stream module
> STREAM_SSL=on: Enable stream_ssl module (SSL req.)
> STREAM_SSL_PREREAD=on: Enable stream_ssl_preread module (SSL req.)
> THREADS=on: Enable threads support
> WWW=on: Enable html sample files
>
> Which is all default for the port, except I also enabled MAIL_* ones as I'll be needing some mail proxying. But at the moment I have no mail {} blocks defined. Looking at these I guess I could trim down defaults. Who needs FLV nowadays :)
>
> I'm not sure which are or aren't 3rd party, but if the descriptions are fully correct, then it looks like I'm not using any "3rd party" ones, because we have options that explicitly state when a module id "3rd party" (and the part I'm not sure is if all are listed as such).
>
> However, it's also compiled with DSO=on, and with the above options, it produces these:
>
> /usr/local/libexec/nginx/ngx_mail_module.so
> /usr/local/libexec/nginx/ngx_stream_module.so
>
> None of which I've loaded at the moment.
>
>
>> 6. Is the website in question an enterprise app or something that is internet visible?
>
> The nginx jail is serving numerous PHP sites and a Python web app, each in their own jails. Using fastcgi for php-fpm and uwsgi for python, all over tcp connections between jails.
>
>
>
> --
> Vlad K.
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20170730/e8d483d5/attachment.html>
More information about the nginx
mailing list