Nginx hang and do not respond with large number of network connection in FIN_WAIT state

[gnusys]

I have more info on the system state at the time the CLOSE_WAIT connections
went sky rocketing

Memory
#######################################

KiB Mem : 13174569+total,  8684164 free, 28138264 used, 94923264 buff/cache
KiB Swap:  4194300 total,  4194300 free,        0 used. 86984112 avail Mem
########################################

CPU
#############################################
top - 16:52:02 up 4 days,  5:37,  1 user,  load average: 3.68, 5.14, 6.06
Tasks: 935 total,   3 running, 932 sleeping,   0 stopped,   0 zombie
%Cpu(s): 22.9 us,  8.0 sy,  0.0 ni, 66.2 id,  2.3 wa,  0.0 hi,  0.6 si,  0.0
st
##############################################

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+
COMMAND
3825721 nobody    20   0 3106072   2.4g   3960 R 105.6  1.9   0:57.32 nginx:
worker process


Nginx worker process was using 105% CPU


!---------------------------------- netstat by state
  16234 CLOSE_WAIT
      6 CLOSING
   1195 ESTABLISHED
      5 FIN_WAIT1
     22 FIN_WAIT2
     14 LAST_ACK
     63 LISTEN
    447 SYN_RECV
     10 SYN_SENT
   1875 TIME_WAIT


!---------------------------------- netstat by state
  20139 CLOSE_WAIT
      2 CLOSING
  23482 ESTABLISHED
  17703 FIN_WAIT1
      4 FIN_WAIT2
     19 LAST_ACK
     63 LISTEN
    863 SYN_RECV
     14 SYN_SENT
    130 TIME_WAIT


Nginx Process 
###################################
root     1724190  1.9  1.9 2835060 2555304 ?     Ss   10:17   7:48 nginx:
master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nobody   3825721 10.6  1.9 3106072 2561764 ?     Rl   16:42   1:02  \_
nginx: worker process
nobody   3825723  0.0  1.9 2835220 2549460 ?     S    16:42   0:00  \_
nginx: cache manager process
####################################



As soon as Nginx was stopped and apache was made the frontend web server the
connections changed as below

############
!---------------------------------- netstat by state
      1 CLOSE_WAIT
      2 CLOSING
    389 ESTABLISHED
  35111 FIN_WAIT1
      2 FIN_WAIT2
     44 LAST_ACK
     42 LISTEN
      1 SYN_RECV
     12 SYN_SENT
     86 TIME_WAIT
##################################

FIN_WAIT1  -- was high , but ESTABLISHED and CLOSE_WAIT went normal

The overall CPU usage and memory remained fine and CPU utilization was
normal too


A difference that I can tell with Nginx and httpd on the server is that
Nginx is configured for http/2 while httpd doesn't have http2 module
enabled

The following is how the listen is configured for the default server in
Nginx

server {
        listen x.x.x.x:80 default_server backlog=16384 reuseport deferred;

server {
        listen x.x.x.x:443 default_server ssl backlog=16384 reuseport
deferred;


the individual vhosts have http2 also enabled

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,282613,282639#msg-282639