socket CLOSE_WAIT problem and 404.html

[Maxim Dounin]

Hello!

On Mon, Mar 08, 2010 at 11:16:25PM +0700, Oon Arfiandwi wrote:

> I have a website that hits by automatic scheduled job every hour from
> thousands sources.
> 
> last time, my website is down because of thousands CLOSE_WAIT when I check
> using "netstat -na"
> I already try to debug refer to http://www.ruby-forum.com/topic/204589
> but I don't know what to do with the debug log.
> 
> when I check the "lsof -p <nginx>":
> i found a lots of sockets (CLOSE_WAIT) and file open of my 404.html.
> i think there's a problem with my 404 configuration.
> -----
> nginx   10500 nobody  357r   REG        8,3        141  52396049
> /home/j/html/404.html
> nginx   10500 nobody  359r   REG        8,3        141  52396049
> /home/j/html/404.html
> nginx   10500 nobody  360r   REG        8,3        141  52396049
> /home/j/html/404.html
> nginx   10500 nobody  362u  IPv4  174783995                  TCP
> 70.36.100.46:http->y.net:49333 (CLOSE_WAIT)
> nginx   10500 nobody  363r   REG        8,3        141  52396049
> /home/j/html/404.html
> nginx   10500 nobody  365u  IPv4  174497664                  TCP
> 70.36.100.46:http->y.net:52096 (CLOSE_WAIT)
> nginx   10500 nobody  366r   REG        8,3        141  52396049
> /home/j/html/404.html
> nginx   10500 nobody  369u  IPv4  174905532                  TCP
> 70.36.100.46:http->y.net:50057 (CLOSE_WAIT)
> nginx   10500 nobody  370r   REG        8,3        141  52396049
> /home/j/html/404.html
> nginx   10500 nobody  371u  IPv4  174729756                  TCP
> 70.36.100.46:http->y.net:33731 (CLOSE_WAIT)
> -----
> 
> 
> here is my "nginx -V"
> -----
> nginx version: nginx/0.7.65
> built by gcc 4.1.2 20080704 (Red Hat 4.1.2-46)
> TLS SNI support disabled
> configure arguments: --with-debug --with-http_ssl_module
> --pid-path=/var/run/nginx.pid --without-mail_smtp_module
> --without-mail_imap_module --without-mail_pop3_module
> --add-module=../nginx_upload_module-2.0.10/
> -----

Some questions in mostly random order:

Is nginx still answers requests or it's stuck completely?

If it's stuck - where?  What nginx workers do?  You should be able 
to find it out by attaching to them with gdb and asking for 
backtrace.

Which event method do you use?  It should be logged at 'notice' 
level on nginx startup.  

Are you able to reproduce problem?  If yes, please try if you 
are able to reproduce it without third party modules/patches.

> here is my config:
> -----
> worker_processes  4;
> error_log  logs/error.log  debug;

Ok, you swithed on debug log here...

[...]

> server {
>     listen       80;
>     server_name  jj.x.y.z jj.x.y;
> 
>     access_log /usr/local/nginx/logs/jj-access.log;
>     error_log  /usr/local/nginx/logs/jj-error.log;

... oops, you swithed it off for the server in question (default 
logging level is 'error').

If you want to obtain usable debug log you should configure it at 
global level and avoid overriding at other levels.  Though in this 
case I believe gdb will help better.

Maxim Dounin