worker_connections are not enough, reusing connections with idle workers

[Maxim Dounin]

Hello!

On Tue, Jun 07, 2022 at 01:18:36PM -0700, Roger Fischer wrote:

> My assumption is that the client requests will be distributed 
> over the 24 worker processes. So no individual worker should 
> come anywhere close to 1000 connections.
> 
> But when I look at the process stats for the workers (ps 
> command), I see a uneven distribution of CPU time used. Note 
> that this is from a different run than the above logs.
> UID        PID  PPID  C STIME TTY          TIME CMD
> netskrt  16905 16902  2 12:19 ?        00:07:05 nginx: worker process
> netskrt  16906 16902  1 12:19 ?        00:04:29 nginx: worker process
> netskrt  16908 16902  1 12:19 ?        00:03:30 nginx: worker process
> netskrt  16910 16902  0 12:19 ?        00:02:26 nginx: worker process
> netskrt  16911 16902  0 12:19 ?        00:01:32 nginx: worker process
> netskrt  16912 16902  0 12:19 ?        00:00:51 nginx: worker process
> netskrt  16913 16902  0 12:19 ?        00:00:11 nginx: worker process
> netskrt  16914 16902  0 12:19 ?        00:00:04 nginx: worker process
> netskrt  16915 16902  0 12:19 ?        00:00:25 nginx: worker process
> netskrt  16916 16902  0 12:19 ?        00:00:01 nginx: worker process
> netskrt  16917 16902  0 12:19 ?        00:00:00 nginx: worker process
> ...
> 
> Is there anything we can configure to more evenly distribute the 
> connections?

As already recommended, consider upgrading to nginx 1.21.6 or 
1.22.0.  There is a known issue in older nginx versions with 
distribution of connections among worker processes on modern Linux 
kernels, fixed in nginx 1.21.6 (http://nginx.org/en/CHANGES):

    *) Bugfix: when using EPOLLEXCLUSIVE on Linux client connections were
       unevenly distributed among worker processes.

In older versions, configuring "accept_mutex on;" or "listen ...  
reuseport;" can be used to improve the distribution of connections 
between worker processes, see 
https://trac.nginx.org/nginx/ticket/2285 for details.

-- 
Maxim Dounin
http://mdounin.ru/