Worker other than the master is listening to the socket

[Maxim Dounin]

Hello!

On Fri, Apr 19, 2019 at 03:16:14AM -0400, allenhe wrote:

> Nginx version: 1.13.6.1
> 
> 1) In our use case, the Nginx is reloaded constantly. you will see lots
> worker process hanging at "nginx: worker process is shutting down" after
> couple days: 
> 
>      58 root       0:00 nginx: master process ./openresty/nginx/sbin/nginx
> -p /opt/applicatio
>  1029 nobody     0:22 nginx: worker process is shutting down
>  1030 nobody     0:27 nginx: worker process is shutting down
>  1041 nobody     0:54 nginx: worker process is shutting down
>  1054 nobody     0:37 nginx: worker process is shutting down
>  1131 nobody     0:02 nginx: worker process is shutting down
>  1132 nobody     0:02 nginx: worker process is shutting down
>  1215 nobody     0:50 nginx: worker process is shutting down
>  1216 nobody     0:53 nginx: worker process is shutting down
>  1515 nobody     0:23 nginx: worker process is shutting down
>  1516 nobody     0:47 nginx: worker process is shutting down
>  1533 nobody     0:03 nginx: worker process is shutting down
>  1534 nobody     0:16 nginx: worker process is shutting down
>  1598 nobody     0:00 nginx: worker process
>  1599 nobody     0:00 nginx: worker process
> 
> 
> 2) And if you now check some listen port on the host using netstat, you will
> see it is owned by the worker process:
> 
> [root at paas-controller-177-1-1-137:~]$ netstat -anp |grep 10080
> tcp        0      0 0.0.0.0:10080           0.0.0.0:*               LISTEN  
>    6288/nginx: worker
> tcp        0      0 10.47.205.136:10080     10.47.205.137:50827    
> ESTABLISHED 6296/nginx: worker
> tcp        0      0 10.47.205.136:10080     10.47.205.137:50833    
> ESTABLISHED 6300/nginx: worker
> tcp        0      0 10.47.205.136:10080     10.47.205.137:49411    
> ESTABLISHED 6296/nginx: worker
> tcp        0      0 10.47.205.136:10080     10.40.157.154:54074    
> ESTABLISHED 6296/nginx: worker
> tcp        0      0 10.47.205.136:10080     10.47.205.137:49715    
> ESTABLISHED 6299/nginx: worker
> tcp6       0      0 :::10080                :::*                    LISTEN  
>    6288/nginx: worker

The "netstat -p" command prints only one process - the one with 
lowest PID - which has the socket open.  But in fact listening 
sockets are open in nginx master process and all worker processes.

After a number of reloads a worker process may get PID 
lower than PID of the master, and "netstat -p" will start to 
report this worker as the process the socket belongs to.  This is 
not something to be afraid of, it is just a result of a "netstat -p" 
limited interface.

To see all processes which have the socket open, consider using 
"ss -nltp" instead.

> 3) So far, you would say it is not a big deal as long as the worker could
> serve the request correctly. but it does NOT. suppose the listen port 10080
> above would proxy the request to the upstream server A initially (and for
> sure this port was listening by the master process), days later, it has been
> changed to the server B and of course the Nginx was been reloaded at the
> same time (and perhaps some worker was left in " is shutting down" state at
> that time). well now, I see this port is listening by a worker process, and
> it would proxy the request to the old server A.

This is not something that can happen, since old worker 
processes, which are in the "shutting down" state, no longer 
accept new connections.  Such processes close all listening 
sockets as soon they are asked to exit.

-- 
Maxim Dounin
http://mdounin.ru/