Identifying "Writing" connections in status stub

[Vlad K.]

On 2017-07-30 11:26, Peter Booth wrote:
> I just reread the thread and realize that you answered q2, and that
> makes the graph even more
> surprising. You say that it son FreeBSD - does this mean that you
> don’t have /proc available to you?
> Is there a procstat or other way to see the equivalent of
> /proc/<pid>/fd - a list of all open file descriptions for a specific
> pid?

procfs is technically available but I'm not enabling it. We do have 
sockstat and fstat though that basically can show the same. I have 
checked and I don't see any corresponding open connections, that's why I 
was wondering about how to list what nginx thinks are connections being 
Written to, so I could find what those connections did last in the 
access log.

I ran a periodic netstat, sorted the IPs (not removing src ports), and 
ran a diff from previous run, keeping only those that aren't +/-. Over 
some period of time, the only connections that stayed open for long were 
actively using the roundcube webmail which (especially with our 
keepalive and http2 enabled) runs a request every minute. But the number 
of those did not correspond to the number reported as Writing. Plus 
these connections have daily highs and downs. "Writing" in the nginx 
status does not correlate with the rest.


> 1. When is the last time that your production nginx was restarted?

In my first post in this thread is a link to the graph (by Munin) where 
it shows a dip in Writing, that's where I restarted. Reloading doesn't 
change the number of Writing reported by nginx.


> 2. Do you have regular restarts?

No, just regular reloads.


> 3. Is there an obstacle to restarting at some point?

If you're asking me if I can restart nginx to check something, I can do 
that.


> 4. Is this a single instance or do you have multiple nginx hosts?

Single instance, one master and one worker thread, in a jail, and 
there's no other jail running nginx on the server.


> 5. What 3rd party models are you using?

These are the options/modules ENABLED for the port:

      DSO=on: Enable dynamic modules support
      FILE_AIO=on: Enable file aio
      HTTP=on: Enable HTTP module
      HTTPV2=on: Enable HTTP/2 protocol support (SSL req.)
      HTTP_ADDITION=on: Enable http_addition module
      HTTP_AUTH_REQ=on: Enable http_auth_request module
      HTTP_CACHE=on: Enable http_cache module
      HTTP_DAV=on: Enable http_webdav module
      HTTP_FLV=on: Enable http_flv module
      HTTP_GUNZIP_FILTER=on: Enable http_gunzip_filter module
      HTTP_GZIP_STATIC=on: Enable http_gzip_static module
      HTTP_MP4=on: Enable http_mp4 module
      HTTP_RANDOM_INDEX=on: Enable http_random_index module
      HTTP_REALIP=on: Enable http_realip module
      HTTP_REWRITE=on: Enable http_rewrite module
      HTTP_SECURE_LINK=on: Enable http_secure_link module
      HTTP_SLICE=on: Enable http_slice module
      HTTP_SSL=on: Enable http_ssl module
      HTTP_STATUS=on: Enable http_stub_status module
      HTTP_SUB=on: Enable http_sub module
      IPV6=on: Enable IPv6 support
      MAIL=on: Enable IMAP4/POP3/SMTP proxy module
      MAIL_IMAP=on: Enable IMAP4 proxy module
      MAIL_POP3=on: Enable POP3 proxy module
      MAIL_SMTP=on: Enable SMTP proxy module
      MAIL_SSL=on: Enable mail_ssl module
      STREAM=on: Enable stream module
      STREAM_SSL=on: Enable stream_ssl module (SSL req.)
      STREAM_SSL_PREREAD=on: Enable stream_ssl_preread module (SSL req.)
      THREADS=on: Enable threads support
      WWW=on: Enable html sample files

Which is all default for the port, except I also enabled MAIL_* ones as 
I'll be needing some mail proxying. But at the moment I have no mail {} 
blocks defined. Looking at these I guess I could trim down defaults. Who 
needs FLV nowadays :)

I'm not sure which are or aren't 3rd party, but if the descriptions are 
fully correct, then it looks like I'm not using any "3rd party" ones, 
because we have options that explicitly state when a module id "3rd 
party" (and the part I'm not sure is if all are listed as such).

However, it's also compiled with DSO=on, and with the above options, it 
produces these:

      /usr/local/libexec/nginx/ngx_mail_module.so
      /usr/local/libexec/nginx/ngx_stream_module.so

None of which I've loaded at the moment.


> 6. Is the website in question an enterprise app or something that is 
> internet visible?

The nginx jail is serving numerous PHP sites and a Python web app, each 
in their own jails. Using fastcgi for php-fpm and uwsgi for python, all 
over tcp connections between jails.



-- 
Vlad K.