Worker other than the master is listening to the socket

[allenhe]

Nginx version: 1.13.6.1

1) In our use case, the Nginx is reloaded constantly. you will see lots
worker process hanging at "nginx: worker process is shutting down" after
couple days: 

     58 root       0:00 nginx: master process ./openresty/nginx/sbin/nginx
-p /opt/applicatio
 1029 nobody     0:22 nginx: worker process is shutting down
 1030 nobody     0:27 nginx: worker process is shutting down
 1041 nobody     0:54 nginx: worker process is shutting down
 1054 nobody     0:37 nginx: worker process is shutting down
 1131 nobody     0:02 nginx: worker process is shutting down
 1132 nobody     0:02 nginx: worker process is shutting down
 1215 nobody     0:50 nginx: worker process is shutting down
 1216 nobody     0:53 nginx: worker process is shutting down
 1515 nobody     0:23 nginx: worker process is shutting down
 1516 nobody     0:47 nginx: worker process is shutting down
 1533 nobody     0:03 nginx: worker process is shutting down
 1534 nobody     0:16 nginx: worker process is shutting down
 1598 nobody     0:00 nginx: worker process
 1599 nobody     0:00 nginx: worker process


2) And if you now check some listen port on the host using netstat, you will
see it is owned by the worker process:

[root at paas-controller-177-1-1-137:~]$ netstat -anp |grep 10080
tcp        0      0 0.0.0.0:10080           0.0.0.0:*               LISTEN  
   6288/nginx: worker
tcp        0      0 10.47.205.136:10080     10.47.205.137:50827    
ESTABLISHED 6296/nginx: worker
tcp        0      0 10.47.205.136:10080     10.47.205.137:50833    
ESTABLISHED 6300/nginx: worker
tcp        0      0 10.47.205.136:10080     10.47.205.137:49411    
ESTABLISHED 6296/nginx: worker
tcp        0      0 10.47.205.136:10080     10.40.157.154:54074    
ESTABLISHED 6296/nginx: worker
tcp        0      0 10.47.205.136:10080     10.47.205.137:49715    
ESTABLISHED 6299/nginx: worker
tcp6       0      0 :::10080                :::*                    LISTEN  
   6288/nginx: worker


3) So far, you would say it is not a big deal as long as the worker could
serve the request correctly. but it does NOT. suppose the listen port 10080
above would proxy the request to the upstream server A initially (and for
sure this port was listening by the master process), days later, it has been
changed to the server B and of course the Nginx was been reloaded at the
same time (and perhaps some worker was left in " is shutting down" state at
that time). well now, I see this port is listening by a worker process, and
it would proxy the request to the old server A.
I suspected this could be caused by the "shutting down" worker processes, so
I "kill -9" all these ones and try again, but nothing changed. even
reloading the Nginx did no help. I shut it down and rerun the binary,
finally this worked and the request can be proxyed to the server B.
I guess maybe some cache in Nginx was doing the bad, any clue?

Thanks,
Allen

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,283700,283856#msg-283856