CLOSE_WAIT sockets piling up on server side

[Igor Sysoev]

On Tue, Aug 14, 2007 at 01:52:44AM -0700, Mansoor Peerbhoy wrote:

> I'm facing a problem while running NGINX IMAP proxy in front of our company-developed IMAP server.
> We're using nginx based off 0.5.20, and the problem is that after a few days of NGINX running, 
> we see that the sockets on our backend IMAP server side are stuck in the CLOSE_WAIT state.
> 
> This socket represents the connection between NGINX and the backend server.
> 
> Basically, if the backend server is running on address X.X.X.X port 1143, and if NGINX is running on interface N.N.N.N,
> then over time, netstat shows me something like:
> 
> Active Internet connections (w/o servers)
> Proto Recv-Q Send-Q Local Address               Foreign Address             State      
> ...
> tcp        1      0 X.X.X.X:1143     N.N.N.N:33820    CLOSE_WAIT
> tcp        1      0 X.X.X.X:1143     N.N.N.N:38940    CLOSE_WAIT
> ...
> 
> And many other such sockets in the CLOSE_WAIT state.
> 
> Now, according to the TCP state transition diagram, a socket goes into CLOSE_WAIT when it has received a FIN from the remote end
> (i.e. the remote end has performed a shutdown or close), but the local end has not yet closed the socket.
> 
> So in this case, it would appear that NGINX is initiating a close, but the backend server side hasn't yet closed its end of the 
> connected socket.
> 
> These CLOSE_WAIT sockets pile up over time, and eventually, NGINX is not able to connect to the backend server anymore.
> 
> Also, since the Recv-Q column of the netstat output indicates that one byte has not yet been read off the server side socket, this
> is quite strange.
> 
> Have you seen problems like this ? I don't yet have access to our IMAP server source code, so I'm not sure why the server side hasn't 
> acknowledged the NGINX closing the socket.
> 
> On the other hand, this could also be due to a misbehaving client that has connected to NGINX and is causing NGINX to initiate a connection close when the server does not expect it. 
> 
> However, I tried manually connecting to our backend IMAP server via. netcat and closing the IMAP connection during various stages of the conversation. 
> 
> In all cases, the server responds to a client closing the socket, by closing its own end of the socket, and therefore, the CLOSE_WAIT state does not persist, and all is fine.
> 
> Our server side is written in Java, but I'm not aware of the lower level APIs that it uses for socket IO.
> 
> Any ideas ?

I suspect that backend is Linux 2.6.x or latest 2.4.x.
There are similar reports on these kernels.


-- 
Igor Sysoev
http://sysoev.ru/en/