Lots of CLOSE_WAIT sockets, nginx+php (WordPress site)

[Maxim Dounin]

Hello!

On Mon, Feb 22, 2010 at 07:46:24AM +0100, Vicente Aguilar wrote:

> >> We're going to start using WP-mu at work so I've been looking 
> >> into it lately and the problem seems to be browser-server 
> >> connections stuck on the CLOSE_WAIT state. With netstat -nap 
> >> I get loads of these:
> >> 
> >> $ netstat -nap | grep CLOSE_WAIT
> >> tcp        1      0 10.10.10.10:80        1.2.3.4:52132     
> >> CLOSE_WAIT  27672/nginx: worker
> > 
> > [...]
> > 
> > What does nginx -V show?  What's in config?
> 
> 
> You're right, should have started there. Sorry. :-)
> 
> $ nginx -V
> nginx version: nginx/0.7.65
> TLS SNI support enabled
> configure arguments: --conf-path=/etc/nginx/nginx.conf 
> --error-log-path=/var/log/nginx/error.log 
> --pid-path=/var/run/nginx.pid --lock-path=/var/lock/nginx.lock 
> --http-log-path=/var/log/nginx/access.log 
> --http-client-body-temp-path=/var/lib/nginx/body 
> --http-proxy-temp-path=/var/lib/nginx/proxy 
> --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --with-debug 
> --with-http_stub_status_module --with-http_flv_module 
> --with-http_ssl_module --with-http_dav_module 
> --with-http_gzip_static_module --with-http_sub_module 
> --with-mail --with-mail_ssl_module --with-ipv6 
> --with-http_perl_module 
> --add-module=/usr/src/debian/nginx/nginx-0.7.65/modules/nginx-upstream-fair 
> --add-module=/usr/src/debian/nginx/nginx-0.7.65/modules/ngx_http_upstream_memcached_hash_module-0.04 
> --add-module=/usr/src/debian/nginx/nginx-0.7.65/modules/ngx_http_secure_download 
> --with-http_proxy_s3_auth

Try compiling without third party modules and patches and check if 
you are able to reproduce the problem.  But see below for more 
simple test to do before this one.

[...]

>         fastcgi_ignore_client_abort on;
>         fastcgi_connect_timeout 60;
>         fastcgi_send_timeout 180;
>         fastcgi_read_timeout 180;

You are ignoring client aborts, and has relatively large timeouts 
set for fastcgi.  Are you sure the connections in question aren't 
disappear as soon as your fastcgi backend finishes preparing 
response?  I.e. check if any particular connection stay for at 
least 5 minutes or so.

Additionally, check if you are able to reproduce the problem with 
fastcgi_ignore_client_abort off.

[...]

> Have tried with several different combinations of 
> worker_processes, sendfile, nopush, keepalive_* ... to no avail. 
> Sometimes it takes longer to hang, but it always end up not 
> responding with > 100 connections in CLOSE_WAIT. Killing nginx, 
> waiting a couple of seconds for the connections in CLOSE_WAIT to 
> disappear from netstat and starting nginx again fixes the issue. 
> No need to restart the PHP processes.

Not responding just because of 100 connections seems strange for 
nginx even with worker_connections 1024, so I suspect you just run 
out of php processes and CLOSE_WAIT's are because of 
fastcgi_ignore_client_abort.

Maxim Dounin