[PATCH] Mail: add the "reuseport" option of the "listen" directive

[Honza Prachař]

On Tue 17. 8. 2021 at 15:49, Maxim Konovalov <maxim at nginx.com> wrote:

> Hi Rob,
>
> On 17.08.2021 06:35, Robert Mueller wrote:
> > # HG changeset patch
> > # User Rob Mueller <robm at fastmail.fm>
> > # Date 1629171218 14400
> > #      Mon Aug 16 23:33:38 2021 -0400
> > # Node ID 89ff95b268e9817b344447b7e6785354229a4bab
> > # Parent  dda421871bc213dd2eb3da0015d6228839323583
> > Mail: add the "reuseport" option of the "listen" directive
> >
> > The "reuseport" option was added to the "listen" directive of the http
> > and stream modules in 1.9.1, but it wasn't added to the mail module. This
> > adds the option to the mail module to make it consistent with the http
> > and stream modules.
> >
> > In newer linux kernel versions (somewhere between 4.9 and 5.10) this
> > option seems much more important. On production debian servers, not
> having
> > or using this option caused processes to become very unbalanced. With 8
> > worker processes, we would see one worker process accepting 70%+ of all
> > connections, a second process with about 10% or so, and the remaining
> > 20% of connections spread over the other 6 processes.  This obviously
> > started causing problems as the worker process accepting the majority
> > of connections would end up being 100% CPU bound well before the servers
> > overall capacity.
> >
> > Adding and enabling this option fixed this entirely, and now all
> > worker processes seem to accept and even spread of connections.
> >
> First, thanks for the patch.
>
> While the reuseport could cure (or hide if you will) the unbalancing you
> see it makes sense to get better understanding what exactly is going on.
>  So far we haven't seen such weird behaviour ourself neither received
> reports about such uneven connections distribution among nginx workers.


Hello!

It looks exactly like known linux epoll behavior, which is nicely explained
here:
https://blog.cloudflare.com/the-sad-state-of-linux-socket-balancing/

Best, Jan Prachař


>
> Any chances you have accept_mutex and/or multi_accept?  Any other ideas?
>
> --
> Maxim Konovalov
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20210818/8d138b71/attachment.htm>