CLOSE_WAIT sockets piling up on server side

[Igor Sysoev]

On Tue, Aug 14, 2007 at 02:10:20AM -0700, Mansoor Peerbhoy wrote:

> [root at imap ~]# uname -a 
> Linux imap.company.com 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:56:28 EST 2006 x86_64 x86_64 x86_64 GNU/Linux
> [root at imap ~]#
> 
> 
> The odd thing is that when we use perdition, we don't see this happening.
> Any clues ?

I do not know. These reports was on HTTP servers, not mail proxy.

> ----- Original Message -----
> From: "Igor Sysoev" <is at rambler-co.ru>
> To: nginx at sysoev.ru
> Sent: Tuesday, August 14, 2007 2:33:10 PM (GMT+0530) Asia/Calcutta
> Subject: Re: CLOSE_WAIT sockets piling up on server side
> 
> On Tue, Aug 14, 2007 at 01:52:44AM -0700, Mansoor Peerbhoy wrote:
> 
> > I'm facing a problem while running NGINX IMAP proxy in front of our company-developed IMAP server.
> > We're using nginx based off 0.5.20, and the problem is that after a few days of NGINX running, 
> > we see that the sockets on our backend IMAP server side are stuck in the CLOSE_WAIT state.
> > 
> > This socket represents the connection between NGINX and the backend server.
> > 
> > Basically, if the backend server is running on address X.X.X.X port 1143, and if NGINX is running on interface N.N.N.N,
> > then over time, netstat shows me something like:
> > 
> > Active Internet connections (w/o servers)
> > Proto Recv-Q Send-Q Local Address               Foreign Address             State      
> > ...
> > tcp        1      0 X.X.X.X:1143     N.N.N.N:33820    CLOSE_WAIT
> > tcp        1      0 X.X.X.X:1143     N.N.N.N:38940    CLOSE_WAIT
> > ...
> > 
> > And many other such sockets in the CLOSE_WAIT state.
> > 
> > Now, according to the TCP state transition diagram, a socket goes into CLOSE_WAIT when it has received a FIN from the remote end
> > (i.e. the remote end has performed a shutdown or close), but the local end has not yet closed the socket.
> > 
> > So in this case, it would appear that NGINX is initiating a close, but the backend server side hasn't yet closed its end of the 
> > connected socket.
> > 
> > These CLOSE_WAIT sockets pile up over time, and eventually, NGINX is not able to connect to the backend server anymore.
> > 
> > Also, since the Recv-Q column of the netstat output indicates that one byte has not yet been read off the server side socket, this
> > is quite strange.
> > 
> > Have you seen problems like this ? I don't yet have access to our IMAP server source code, so I'm not sure why the server side hasn't 
> > acknowledged the NGINX closing the socket.
> > 
> > On the other hand, this could also be due to a misbehaving client that has connected to NGINX and is causing NGINX to initiate a connection close when the server does not expect it. 
> > 
> > However, I tried manually connecting to our backend IMAP server via. netcat and closing the IMAP connection during various stages of the conversation. 
> > 
> > In all cases, the server responds to a client closing the socket, by closing its own end of the socket, and therefore, the CLOSE_WAIT state does not persist, and all is fine.
> > 
> > Our server side is written in Java, but I'm not aware of the lower level APIs that it uses for socket IO.
> > 
> > Any ideas ?
> 
> I suspect that backend is Linux 2.6.x or latest 2.4.x.
> There are similar reports on these kernels.
> 
> 
> -- 
> Igor Sysoev
> http://sysoev.ru/en/
> 
> 
> 

-- 
Igor Sysoev
http://sysoev.ru/en/