Nginx hang and do not respond with large number of network connection in FIN_WAIT state

[Maxim Dounin]

Hello!

On Thu, Jan 10, 2019 at 08:27:08AM +0530, Anoop Alias wrote:

> Have had a really strange issue on a Nginx server configured as a reverse
> proxy wherein the server stops responding when the network connections in
> ESTABLISHED state and FIN_WAIT state in very high compared to normal
> working
> 
> If you see the below network graph, at around 00:30 hours there is a big
> spike in network connections in FIN_WAIT state, to around 12000 from the
> normal value of  ~20
> 
> https://i.imgur.com/wb6VMWo.png
> 
> At this state, Nginx stops responding fully and does not work even after a
> full restart of the service.

>From the image it looks like there are CLOSE_WAIT sockets, not 
FIN_WAIT.  Most likely this means that nginx was blocked on 
something - probably system resources somewhere inside the kernel, 
or disk operation (on a dead disk?), or something like this.  
E.g., this is something trivial to observe if you are serving 
files from an NFS share, and the NFS server dies.

Alternatively, this might be a bug somewhere.  In particular, it 
looks like you are using various 3rd party modules, and a bug in 
any of them can cause similar problems.  But I don't really think 
this is the case, as restarting nginx usually fixes such problems.

-- 
Maxim Dounin
http://mdounin.ru/