Lots of CLOSE_WAIT sockets, nginx+php (WordPress site)
Maxim Dounin
mdounin at mdounin.ru
Thu Feb 25 15:59:21 MSK 2010
Hello!
On Thu, Feb 25, 2010 at 10:11:23AM +0100, Vicente Aguilar wrote:
> Hi
>
> > There will be another connection between nginx and php, but it
> > will be live (in ESTABLISHED state). It will wait for php to
> > finish request processing and sending response back to nginx.
> > Once php is done - nginx will close both nginx-php and
> > client-nginx connections.
>
>
> That's what I though, but I've got no nginx-php connections in ESTABLISHED mode when a client-nginx connection goes to CLOSE_WAIT.
>
> This is what I get now on a netstat -nap:
>
> Active Internet connections (servers and established)
> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
> tcp 0 0 127.0.0.1:9000 0.0.0.0:* LISTEN 13040/php5-fpm
> tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN 6731/mysqld
> tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 16936/nginx
> tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 881/sshd
> tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1416/exim4
> tcp 324 0 10.10.10.10:80 66.249.68.247:36979 ESTABLISHED -
> tcp 113 0 127.0.0.1:80 127.0.0.1:39229 CLOSE_WAIT -
> tcp 335 0 10.10.10.10:80 65.55.207.102:47057 ESTABLISHED -
> tcp 116 0 10.10.10.10:80 83.170.113.102:60149 ESTABLISHED -
> tcp 413 0 10.10.10.10:80 81.52.143.26:51658 CLOSE_WAIT -
> tcp 117 0 10.10.10.10:80 83.170.113.102:56117 CLOSE_WAIT -
> tcp 1 0 10.10.10.10:80 66.249.68.247:62666 CLOSE_WAIT 16940/nginx: worker
> tcp 1 0 10.10.10.10:80 66.249.68.247:37085 CLOSE_WAIT 16939/nginx: worker
> tcp 1 0 10.10.10.10:80 66.249.68.247:51700 CLOSE_WAIT 16938/nginx: worker
> tcp 1 0 10.10.10.10:80 67.195.115.83:53503 CLOSE_WAIT 16937/nginx: worker
> tcp 117 0 10.10.10.10:80 74.52.50.50:52833 CLOSE_WAIT -
> tcp 483 0 10.10.10.10:80 213.171.250.126:35648 ESTABLISHED -
> tcp 0 288 10.10.10.10:22 193.145.230.6:50184 ESTABLISHED 17110/0
> tcp 1 0 10.10.10.10:80 87.68.237.93:51575 CLOSE_WAIT 16939/nginx: worker
> tcp 244 0 10.10.10.10:80 123.125.66.48:22675 CLOSE_WAIT -
> tcp 1 0 10.10.10.10:80 193.145.230.6:50134 CLOSE_WAIT 16938/nginx: worker
> tcp 116 0 10.10.10.10:80 74.52.50.50:55178 ESTABLISHED -
> tcp6 0 0 :::22 :::* LISTEN 881/sshd
> udp 0 0 10.10.10.10:53 0.0.0.0:* 1515/tinydns
> udp 0 0 0.0.0.0:68 0.0.0.0:* 769/dhclient3
> Active UNIX domain sockets (servers and established)
> Proto RefCnt Flags Type State I-Node PID/Program name Path
> unix 2 [ ACC ] STREAM LISTENING 481377 6731/mysqld /var/run/mysqld/mysqld.sock
> unix 3 [ ] DGRAM 601444 867/rsyslogd /dev/log
> unix 2 [ ] DGRAM 673 298/udevd @/org/kernel/udev/udevd
> unix 2 [ ] DGRAM 611850 17110/0
> unix 3 [ ] STREAM CONNECTED 609800 16936/nginx
> unix 3 [ ] STREAM CONNECTED 609799 16936/nginx
> unix 3 [ ] STREAM CONNECTED 609797 16936/nginx
> unix 3 [ ] STREAM CONNECTED 609796 16936/nginx
> unix 3 [ ] STREAM CONNECTED 609794 16936/nginx
> unix 3 [ ] STREAM CONNECTED 609793 16936/nginx
> unix 3 [ ] STREAM CONNECTED 609791 16936/nginx
> unix 3 [ ] STREAM CONNECTED 609790 16936/nginx
> unix 2 [ ] DGRAM 533124 769/dhclient3
> unix 2 [ ] DGRAM 481379 6735/logger
> unix 3 [ ] STREAM CONNECTED 324617 27662/php5-fpm
> unix 3 [ ] STREAM CONNECTED 324616 27662/php5-fpm
> unix 3 [ ] STREAM CONNECTED 324613 27662/php5-fpm
> unix 3 [ ] STREAM CONNECTED 324612 27662/php5-fpm
>
>
> 11 client-nginx connections in CLOSE_WAIT, no nginx-php
> connections. Unless they're the last four unix domain sockets
> connections, but I've configured nginx to use 127.0.0.1:9000 as
> the fcgi server.
>From the output you provided it looks like all nginx workers are
locked out, either doing something or waiting for some system
resources. As you can see - all connections accepted by nginx (6
connections which have nginx process listed in pid column) are in
CLOSE_WAIT state, and there are other connections to port 80 which
are sitting in listen queue. Am I right in the assumption that
nginx does not answer any requests?
You have to examine nginx workers to find out what they are doing.
Try starting from top, truss, gdb and examining your system logs.
Note well: you haven't posted full config you use, so please check
yourself for possible loops in it. I've recently posted some
patches which take care of several loops which aren't automatically
resolved now, see here for patch and example loops:
http://nginx.org/pipermail/nginx-devel/2010-January/000099.html
It should be trivial to find if it's the cause though, as nginx
worker will eat 100% cpu once caught in such loop.
Note well 2: I've already asked you to try compiling without third
party modules and patches and check if you are able to reproduce
the problem. It doesn't really make sense to proceed any further
without doing this.
> How can I debug what these CLOSE_WAIT connections were doing,
> which request were they serving? Anything I can activate on the
> logs or on nginx-status, a la Apache's extended server-status?
You have to enable debug log (see
http://nginx.org/en/docs/debugging_log.html). Then it will be
possible to map fd number to the particular request (and it's full
logs). Under linux it should be possible to find out fd number of
the particular connection via lsof -p <pid-of-nginx-worker>.
Maxim Dounin
More information about the nginx
mailing list