Nginx hang and do not respond with large number of network connection in FIN_WAIT state
mdounin at mdounin.ru
Thu Jan 10 13:45:31 UTC 2019
On Thu, Jan 10, 2019 at 08:27:08AM +0530, Anoop Alias wrote:
> Have had a really strange issue on a Nginx server configured as a reverse
> proxy wherein the server stops responding when the network connections in
> ESTABLISHED state and FIN_WAIT state in very high compared to normal
> If you see the below network graph, at around 00:30 hours there is a big
> spike in network connections in FIN_WAIT state, to around 12000 from the
> normal value of ~20
> At this state, Nginx stops responding fully and does not work even after a
> full restart of the service.
>From the image it looks like there are CLOSE_WAIT sockets, not
FIN_WAIT. Most likely this means that nginx was blocked on
something - probably system resources somewhere inside the kernel,
or disk operation (on a dead disk?), or something like this.
E.g., this is something trivial to observe if you are serving
files from an NFS share, and the NFS server dies.
Alternatively, this might be a bug somewhere. In particular, it
looks like you are using various 3rd party modules, and a bug in
any of them can cause similar problems. But I don't really think
this is the case, as restarting nginx usually fixes such problems.
More information about the nginx