Nginx most connections in FIN_WAIT2 state

[Maxim Dounin]

Hello!

On Wed, Jul 27, 2022 at 05:24:07PM -0400, RasmithaM wrote:

> We are using Nginx for outbound connectivity to client , 
> 
> I see all the requests are going to FIN_WAIT2 state , even server sending us
> the ACK. 
> 
> the fin_timeout is set to 60 sec , but we observed that the process
> continues to stay in FIN_WAIT2 even after 60sec. 
> Is this kernel issue / Nginc issue ?
> netstat -tan | awk '{print $6}' | sort | uniq -c
>    1793 CLOSE_WAIT
>      40 ESTABLISHED
>    6398 FIN_WAIT2
>       1 Foreign
>      22 LISTEN
>     152 TIME_WAIT
>       1 established)
> 
> This is filling up the number of sockets finally have to restart Nginx to
> release the FIN_WAIT2 processes. 

Are you seeing FIN_WAIT2 on connections from nginx to upstream 
servers?  Or on connections from clients to nginx?

The FIN_WAIT2 state suggests that shutdown() or close() was called 
on the socket, so FIN was sent to the other end, and then ACK was 
received.  At this point system is waiting for the other end's 
FIN.  Depending on whether shutdown() or close() was used, it's 
either application (in case of shutdown() and no close()) or 
kernel (in case of close()) responsibility to clean up things.

Given that nginx never uses shutdown() on sockets to upstream 
servers, for connections from nginx to upstream servers the 
only remaining option seems to be a kernel issue.  It is not clear 
why restarting nginx helps to clean up things though.

For connections from clients to nginx, nginx can use shutdown() 
and then keep reading from the socket for up to lingering_time 
(http://nginx.org/r/lingering_time), which is 30 seconds by 
default.  That is, on connections from clients to nginx the 
FIN_WAIT2 state can be seen for up to 90 seconds assuming default 
settings.  If you are seeing sockets in this state for a 
significantly longer time, it might be a good idea to further 
debug what's going on.  In particular, nginx debugging log might 
be helpful (see http://nginx.org/en/docs/debugging_log.html for 
details).

Hope this helps.

-- 
Maxim Dounin
http://mdounin.ru/