worker_connections are not enough, reusing connections with idle workers

[Roger Fischer]

Thanks, Sergey.

We are simulating 1000 clients. Some get cache hits, and some go upstream. So there are more than 1000 connections.

We have 24 workers running, each configured: events { worker_connections  1024; }

We are seeing the following errors from nginx:
[warn] 21151#21151: 1024 worker_connections are not enough, reusing connections
[crit] 21151#21151: accept4() failed (24: Too many open files)
[alert] 21151#21151: *15716 socket() failed (24: Too many open files) while connecting to upstream,

I am assuming the second and third error are for the OS limit. But the first seems to be from a worker process.

My assumption is that the client requests will be distributed over the 24 worker processes. So no individual worker should come anywhere close to 1000 connections.

But when I look at the process stats for the workers (ps command), I see a uneven distribution of CPU time used. Note that this is from a different run than the above logs.
UID        PID  PPID  C STIME TTY          TIME CMD
netskrt  16905 16902  2 12:19 ?        00:07:05 nginx: worker process
netskrt  16906 16902  1 12:19 ?        00:04:29 nginx: worker process
netskrt  16908 16902  1 12:19 ?        00:03:30 nginx: worker process
netskrt  16910 16902  0 12:19 ?        00:02:26 nginx: worker process
netskrt  16911 16902  0 12:19 ?        00:01:32 nginx: worker process
netskrt  16912 16902  0 12:19 ?        00:00:51 nginx: worker process
netskrt  16913 16902  0 12:19 ?        00:00:11 nginx: worker process
netskrt  16914 16902  0 12:19 ?        00:00:04 nginx: worker process
netskrt  16915 16902  0 12:19 ?        00:00:25 nginx: worker process
netskrt  16916 16902  0 12:19 ?        00:00:01 nginx: worker process
netskrt  16917 16902  0 12:19 ?        00:00:00 nginx: worker process
...

Is there anything we can configure to more evenly distribute the connections?

Thanks…

Roger


> On Jun 3, 2022, at 8:40 PM, Sergey A. Osokin <osa at freebsd.org.ru> wrote:
> 
> Hi Roger,
> 
> hope you're doing well.
> 
> On Fri, Jun 03, 2022 at 05:38:07PM -0700, Roger Fischer wrote:
>> Hello,
>> 
>> my understanding is that worker_connections applies to each worker
>> (eg. when set to 1024, 10 worker processes could handle up to 10240
>> connections).
> 
> That's exactly right.  Please read the following link [1] to get more
> details.
> 
>> But we are seeing 1024 worker_connections are not enough, reusing
>> connections from one worker while other workers are idle.
> 
> So, it's possibe to increase the number of those.
> 
>> Is there something we can do to balance connections more evenly
>> across workers?
> 
> Could you please add a bit more details on this.  Please note, that
> there were several improvements on that topic, so please follow the
> recommendations below.
> 
>> nginx version: nginx/1.19.9
> 
> Recent stable version is 1.22.0, [2] so I'd recommend to update to
> that version.
> 
> Thank you.
> 
> References
> 1. https://nginx.org/en/docs/ngx_core_module.html#worker_connections
> 2. http://nginx.org/en/CHANGES-1.22
> 
> --
> Sergey A. Osokin
> _______________________________________________
> nginx mailing list -- nginx at nginx.org
> To unsubscribe send an email to nginx-leave at nginx.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20220607/b07a7793/attachment.htm>