CLOSE_WAIT sockets piling up on server side

[Mansoor Peerbhoy]

Hello Igor

I'm facing a problem while running NGINX IMAP proxy in front of our company-developed IMAP server.
We're using nginx based off 0.5.20, and the problem is that after a few days of NGINX running, 
we see that the sockets on our backend IMAP server side are stuck in the CLOSE_WAIT state.

This socket represents the connection between NGINX and the backend server.

Basically, if the backend server is running on address X.X.X.X port 1143, and if NGINX is running on interface N.N.N.N,
then over time, netstat shows me something like:

Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State      
...
tcp        1      0 X.X.X.X:1143     N.N.N.N:33820    CLOSE_WAIT
tcp        1      0 X.X.X.X:1143     N.N.N.N:38940    CLOSE_WAIT
...

And many other such sockets in the CLOSE_WAIT state.

Now, according to the TCP state transition diagram, a socket goes into CLOSE_WAIT when it has received a FIN from the remote end
(i.e. the remote end has performed a shutdown or close), but the local end has not yet closed the socket.

So in this case, it would appear that NGINX is initiating a close, but the backend server side hasn't yet closed its end of the 
connected socket.

These CLOSE_WAIT sockets pile up over time, and eventually, NGINX is not able to connect to the backend server anymore.

Also, since the Recv-Q column of the netstat output indicates that one byte has not yet been read off the server side socket, this
is quite strange.

Have you seen problems like this ? I don't yet have access to our IMAP server source code, so I'm not sure why the server side hasn't 
acknowledged the NGINX closing the socket.

On the other hand, this could also be due to a misbehaving client that has connected to NGINX and is causing NGINX to initiate a connection close when the server does not expect it. 

However, I tried manually connecting to our backend IMAP server via. netcat and closing the IMAP connection during various stages of the conversation. 

In all cases, the server responds to a client closing the socket, by closing its own end of the socket, and therefore, the CLOSE_WAIT state does not persist, and all is fine.

Our server side is written in Java, but I'm not aware of the lower level APIs that it uses for socket IO.

Any ideas ?


Regards,
Mansoor Peerbhoy