[PATCH] Mail: add the "reuseport" option of the "listen" directive

[Maxim Dounin]

Hello!

On Wed, Aug 18, 2021 at 11:14:25AM +1000, Robert Mueller wrote:

> > First, thanks for the patch.
> > 
> > While the reuseport could cure (or hide if you will) the unbalancing you
> > see it makes sense to get better understanding what exactly is going on.
> >  So far we haven't seen such weird behaviour ourself neither received
> > reports about such uneven connections distribution among nginx workers.
> > 
> > Any chances you have accept_mutex and/or multi_accept?  Any other ideas?
> 
> Unfortunately I'm not 100% sure what's causing it, but it's 
> pretty easy for us to reproduce even on our development 
> machines. Just to show there's no accept_mutex or multi_accept 
> in our config.

[...]

> Now lets change `listen 10.a.b.c:993 ssl reuseport` to `listen 10.a.b.c:993 ssl` and restart.
> 
> ```
> # ps auxw | grep nginx | grep mail
> root      559885  0.0  0.0  68472  3104 ?        Ss   21:01   0:00 nginx: master process /usr/local/nginx/sbin/nginx -c /etc/nginx/mail.conf
> nobody    559886  0.0  0.3  95620 30448 ?        S    21:01   0:00 nginx: worker process /usr/local/nginx/sbin/nginx -c /etc/nginx/mail.conf
> nobody    559887  0.0  0.3  95620 30448 ?        S    21:01   0:00 nginx: worker process /usr/local/nginx/sbin/nginx -c /etc/nginx/mail.conf
> nobody    559888  0.0  0.3  95620 30448 ?        S    21:01   0:00 nginx: worker process /usr/local/nginx/sbin/nginx -c /etc/nginx/mail.conf
> nobody    559889  0.0  0.3  95620 30448 ?        S    21:01   0:00 nginx: worker process /usr/local/nginx/sbin/nginx -c /etc/nginx/mail.conf
> # perl -e 'use IO::Socket::SSL; for (1..1000) { push @s, IO::Socket::SSL->new("imap.foo:993"); } print "done\n"; sleep 1000;'
> done
> ^Z
> [5]+  Stopped
> # for i in 559886 559887 559888 559889; do echo "$i - " `ls /proc/$i/fd | wc -l`; done
> 559886 -  1054
> 559887 -  57
> 559888 -  60
> 559889 -  57
> ```
> 
> And as you can see, a completely uneven distribution of 
> connections between processes! This doesn't just occur on our 
> development machines either (e.g. it's not related to the source 
> IP or anything), it occurs on production systems with 
> connections arriving from real world customers and clients 
> scattered around the world.

Could you please test if compiling with --with-cc-opt="-DNGX_HAVE_EPOLLEXCLUSIVE=0" 
improves things, notably on production systems?  In my limited 
testing it seems to be improve things, and if this is indeed the 
case, we can consider removing use of EPOLLEXCLUSIVE.

Note well that the above test may, and probably will, result in 
uneven distribution in many cases.  Things are, however, expected 
to improve over time, since new connections are more likely to be 
accepted by idle worker processes and not by worker processes 
which are already close to 100% CPU usage.  That is, unless worker
processes are restarted regularly in your setup for some reason, 
you probably should observe balancing improvements over time.  The 
behaviour might depend on EPOLLEXCLUSIVE though, and it would be 
interesting to observe what happens in practice.

[...]

> As you can see, without the reuseport option, this causes severe 
> scalability problems for us.

I tend to think that reuseport is a bad option for load balancing 
between worker processes, as it can be easily tricked by an outside 
actor to select a particular worker process, and this opens an 
obvious DoS attack vector.

-- 
Maxim Dounin
http://mdounin.ru/