CLOSE_WAIT sockets piling up on server side
mansoor at zimbra.com
Tue Aug 14 13:10:20 MSD 2007
[root at imap ~]# uname -a
Linux imap.company.com 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:56:28 EST 2006 x86_64 x86_64 x86_64 GNU/Linux
[root at imap ~]#
The odd thing is that when we use perdition, we don't see this happening.
Any clues ?
----- Original Message -----
From: "Igor Sysoev" <is at rambler-co.ru>
To: nginx at sysoev.ru
Sent: Tuesday, August 14, 2007 2:33:10 PM (GMT+0530) Asia/Calcutta
Subject: Re: CLOSE_WAIT sockets piling up on server side
On Tue, Aug 14, 2007 at 01:52:44AM -0700, Mansoor Peerbhoy wrote:
> I'm facing a problem while running NGINX IMAP proxy in front of our company-developed IMAP server.
> We're using nginx based off 0.5.20, and the problem is that after a few days of NGINX running,
> we see that the sockets on our backend IMAP server side are stuck in the CLOSE_WAIT state.
> This socket represents the connection between NGINX and the backend server.
> Basically, if the backend server is running on address X.X.X.X port 1143, and if NGINX is running on interface N.N.N.N,
> then over time, netstat shows me something like:
> Active Internet connections (w/o servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State
> tcp 1 0 X.X.X.X:1143 N.N.N.N:33820 CLOSE_WAIT
> tcp 1 0 X.X.X.X:1143 N.N.N.N:38940 CLOSE_WAIT
> And many other such sockets in the CLOSE_WAIT state.
> Now, according to the TCP state transition diagram, a socket goes into CLOSE_WAIT when it has received a FIN from the remote end
> (i.e. the remote end has performed a shutdown or close), but the local end has not yet closed the socket.
> So in this case, it would appear that NGINX is initiating a close, but the backend server side hasn't yet closed its end of the
> connected socket.
> These CLOSE_WAIT sockets pile up over time, and eventually, NGINX is not able to connect to the backend server anymore.
> Also, since the Recv-Q column of the netstat output indicates that one byte has not yet been read off the server side socket, this
> is quite strange.
> Have you seen problems like this ? I don't yet have access to our IMAP server source code, so I'm not sure why the server side hasn't
> acknowledged the NGINX closing the socket.
> On the other hand, this could also be due to a misbehaving client that has connected to NGINX and is causing NGINX to initiate a connection close when the server does not expect it.
> However, I tried manually connecting to our backend IMAP server via. netcat and closing the IMAP connection during various stages of the conversation.
> In all cases, the server responds to a client closing the socket, by closing its own end of the socket, and therefore, the CLOSE_WAIT state does not persist, and all is fine.
> Our server side is written in Java, but I'm not aware of the lower level APIs that it uses for socket IO.
> Any ideas ?
I suspect that backend is Linux 2.6.x or latest 2.4.x.
There are similar reports on these kernels.
More information about the nginx