<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><font color="#5c0700" class="">See below</font><div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Jul 30, 2017, at 6:12 AM, Vlad K. <<a href="mailto:nginx-ml@acheronmedia.hr" class="">nginx-ml@acheronmedia.hr</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">On 2017-07-30 11:26, Peter Booth wrote:<br class=""><blockquote type="cite" class="">I just reread the thread and realize that you answered q2, and that<br class="">makes the graph even more<br class="">surprising. You say that it son FreeBSD - does this mean that you<br class="">don’t have /proc available to you?<br class="">Is there a procstat or other way to see the equivalent of<br class="">/proc/<pid>/fd - a list of all open file descriptions for a specific<br class="">pid?<br class=""></blockquote><br class="">procfs is technically available but I'm not enabling it. We do have sockstat and fstat though that basically can show the same. I have checked and I don't see any corresponding open connections, that's why I was wondering about how to list what nginx thinks are connections being Written to, so I could find what those connections did last in the access log.<br class=""></div></div></blockquote><div><br class=""></div><div><font color="#942192" class="">It appears that you have a lot of data that could help in this analysis.</font></div><div><font color="#942192" class="">How frequently is the status page being queried? Does every status datapoint get recorded</font></div><div><font color="#942192" class="">or is munin showing some rolled up rrd data?</font></div><div><font color="#942192" class=""><br class=""></font></div><div><br class=""></div><br class=""><blockquote type="cite" class=""><div class=""><div class=""><br class="">I ran a periodic netstat, sorted the IPs (not removing src ports), and ran a diff from previous run, keeping only those that aren't +/-. Over some period of time, the only connections that stayed open for long were actively using the roundcube webmail which (especially with our keepalive and http2 enabled) runs a request every minute. But the number of those did not correspond to the number reported as Writing. Plus these connections have daily highs and downs. "Writing" in the nginx status does not correlate with the rest.<br class=""></div></div></blockquote><div><br class=""></div><div><font color="#942192" class="">If you open the status page in a browser do the numbers report match what you see with netstat?</font></div><div><br class=""></div><br class=""><blockquote type="cite" class=""><div class=""><div class=""><br class=""><br class=""><blockquote type="cite" class="">1. When is the last time that your production nginx was restarted?<br class=""></blockquote><br class="">In my first post in this thread is a link to the graph (by Munin) where it shows a dip in Writing, that's where I restarted. Reloading doesn't change the number of Writing reported by nginx.<br class=""></div></div></blockquote><div><br class=""></div><div><font color="#942192" class="">Thats what I thought. <b class="">I think that the graph looks weird</b>. Over the time interval [18 July, 23 July] the time series</font></div><div><font color="#942192" class=""> labelled “writing” connections increases almost monotonically from approx 5 to 15. Then the nginx is restarted</font></div><div><font color="#942192" class="">and the graph jumps back unto about 12/13 and continues writing again. Do you have a hypothesis that explains </font></div><div><font color="#942192" class="">why the graph could jump back to 12/13, rather than spend a few days increasing linearly in the way it did from </font></div><div><font color="#942192" class="">the 18th to the 23rd?</font></div><div><font color="#942192" class=""><br class=""></font></div><div><font color="#942192" class="">How long was nginx down for? If you </font><span style="color: rgb(148, 33, 146);" class="">graph only the “writing” variable for just 23rd July does the length of </span></div><div><span style="color: rgb(148, 33, 146);" class="">time that the # of writing connections is thought</span><span style="color: rgb(148, 33, 146);" class="">to be 0 make sense?</span></div><div><br class=""></div><div><font color="#942192" class="">I wonder whether what you are seeing could be a side-effect of the server being in a FreeBSD jail?</font></div><div><font color="#942192" class="">I haven’t used FreeBSD. My understanding is that FreeBSD jails are more than just chroots, similar to Solaris Zones</font></div><div><font color="#942192" class="">or OpenVZ on Linux. Does each jail have separate, independent sysctl settings/ulimits?</font></div><div><font color="#942192" class="">Do any of the other nginx sites in other jails exhibit the same behavior?</font></div><div><font color="#942192" class="">In FreeBSD jails is there an equivalent of Dom) in a XEN hypervisor? A parent or root OS?</font></div><div><font color="#942192" class="">If so, do you see all connections on al jails the you log into it? If wondering if you are hitting some ulimit or </font></div><div><font color="#942192" class="">resource shortage on the host as a whole?</font></div><br class=""><blockquote type="cite" class=""><div class=""><div class=""><br class=""><br class=""><blockquote type="cite" class="">2. Do you have regular restarts?<br class=""></blockquote><br class="">No, just regular reloads.<br class=""><br class=""><br class=""><blockquote type="cite" class="">3. Is there an obstacle to restarting at some point?<br class=""></blockquote><br class="">If you're asking me if I can restart nginx to check something, I can do that.<br class=""><br class=""><br class=""><blockquote type="cite" class="">4. Is this a single instance or do you have multiple nginx hosts?<br class=""></blockquote><br class="">Single instance, one master and one worker thread, in a jail, and there's no other jail running nginx on the server.<br class=""><br class=""><br class=""><blockquote type="cite" class="">5. What 3rd party models are you using?<br class=""></blockquote><br class="">These are the options/modules ENABLED for the port:<br class=""><br class=""> DSO=on: Enable dynamic modules support<br class=""> FILE_AIO=on: Enable file aio<br class=""> HTTP=on: Enable HTTP module<br class=""> HTTPV2=on: Enable HTTP/2 protocol support (SSL req.)<br class=""> HTTP_ADDITION=on: Enable http_addition module<br class=""> HTTP_AUTH_REQ=on: Enable http_auth_request module<br class=""> HTTP_CACHE=on: Enable http_cache module<br class=""> HTTP_DAV=on: Enable http_webdav module<br class=""> HTTP_FLV=on: Enable http_flv module<br class=""> HTTP_GUNZIP_FILTER=on: Enable http_gunzip_filter module<br class=""> HTTP_GZIP_STATIC=on: Enable http_gzip_static module<br class=""> HTTP_MP4=on: Enable http_mp4 module<br class=""> HTTP_RANDOM_INDEX=on: Enable http_random_index module<br class=""> HTTP_REALIP=on: Enable http_realip module<br class=""> HTTP_REWRITE=on: Enable http_rewrite module<br class=""> HTTP_SECURE_LINK=on: Enable http_secure_link module<br class=""> HTTP_SLICE=on: Enable http_slice module<br class=""> HTTP_SSL=on: Enable http_ssl module<br class=""> HTTP_STATUS=on: Enable http_stub_status module<br class=""> HTTP_SUB=on: Enable http_sub module<br class=""> IPV6=on: Enable IPv6 support<br class=""> MAIL=on: Enable IMAP4/POP3/SMTP proxy module<br class=""> MAIL_IMAP=on: Enable IMAP4 proxy module<br class=""> MAIL_POP3=on: Enable POP3 proxy module<br class=""> MAIL_SMTP=on: Enable SMTP proxy module<br class=""> MAIL_SSL=on: Enable mail_ssl module<br class=""> STREAM=on: Enable stream module<br class=""> STREAM_SSL=on: Enable stream_ssl module (SSL req.)<br class=""> STREAM_SSL_PREREAD=on: Enable stream_ssl_preread module (SSL req.)<br class=""> THREADS=on: Enable threads support<br class=""> WWW=on: Enable html sample files<br class=""><br class="">Which is all default for the port, except I also enabled MAIL_* ones as I'll be needing some mail proxying. But at the moment I have no mail {} blocks defined. Looking at these I guess I could trim down defaults. Who needs FLV nowadays :)<br class=""><br class="">I'm not sure which are or aren't 3rd party, but if the descriptions are fully correct, then it looks like I'm not using any "3rd party" ones, because we have options that explicitly state when a module id "3rd party" (and the part I'm not sure is if all are listed as such).<br class=""><br class="">However, it's also compiled with DSO=on, and with the above options, it produces these:<br class=""><br class=""> /usr/local/libexec/nginx/ngx_mail_module.so<br class=""> /usr/local/libexec/nginx/ngx_stream_module.so<br class=""><br class="">None of which I've loaded at the moment.<br class=""><br class=""><br class=""><blockquote type="cite" class="">6. Is the website in question an enterprise app or something that is internet visible?<br class=""></blockquote><br class="">The nginx jail is serving numerous PHP sites and a Python web app, each in their own jails. Using fastcgi for php-fpm and uwsgi for python, all over tcp connections between jails.<br class=""><br class=""><br class=""><br class="">-- <br class="">Vlad K.<br class="">_______________________________________________<br class="">nginx mailing list<br class=""><a href="mailto:nginx@nginx.org" class="">nginx@nginx.org</a><br class="">http://mailman.nginx.org/mailman/listinfo/nginx</div></div></blockquote></div><br class=""></div></body></html>