[PATCH] Mail: add the "reuseport" option of the "listen" directive

[Maxim Konovalov]

Hi Rob,

On 17.08.2021 06:35, Robert Mueller wrote:
> # HG changeset patch
> # User Rob Mueller <robm at fastmail.fm>
> # Date 1629171218 14400
> #      Mon Aug 16 23:33:38 2021 -0400
> # Node ID 89ff95b268e9817b344447b7e6785354229a4bab
> # Parent  dda421871bc213dd2eb3da0015d6228839323583
> Mail: add the "reuseport" option of the "listen" directive
> 
> The "reuseport" option was added to the "listen" directive of the http
> and stream modules in 1.9.1, but it wasn't added to the mail module. This
> adds the option to the mail module to make it consistent with the http
> and stream modules.
> 
> In newer linux kernel versions (somewhere between 4.9 and 5.10) this
> option seems much more important. On production debian servers, not having
> or using this option caused processes to become very unbalanced. With 8
> worker processes, we would see one worker process accepting 70%+ of all
> connections, a second process with about 10% or so, and the remaining
> 20% of connections spread over the other 6 processes.  This obviously
> started causing problems as the worker process accepting the majority
> of connections would end up being 100% CPU bound well before the servers
> overall capacity.
> 
> Adding and enabling this option fixed this entirely, and now all
> worker processes seem to accept and even spread of connections.
> 
First, thanks for the patch.

While the reuseport could cure (or hide if you will) the unbalancing you
see it makes sense to get better understanding what exactly is going on.
 So far we haven't seen such weird behaviour ourself neither received
reports about such uneven connections distribution among nginx workers.

Any chances you have accept_mutex and/or multi_accept?  Any other ideas?

-- 
Maxim Konovalov