[PATCH] Mail: add the "reuseport" option of the "listen" directive

[Maxim Dounin]

Hello!

On Thu, Aug 19, 2021 at 11:23:05AM +1000, Robert Mueller wrote:

> > Since reuseport uses hash of the source address to balance 
> > incoming connections between sockets, the client can choose a 
> > source port to use so the hash will direct the connection to a 
> > particular socket, that is, to a particular worker process.
> 
> But if a client is choosing the same source-ip:source-port, it's 
> not a real client or OS TCP stack, it's some system using raw 
> packets designed for attacking another system, and in that case 
> there's many other attack options available. I'm not convinced 
> this is a large real-world concern for most users of nginx.

It is trivial to select arbitrary source port using normal TCP 
stack.  Even if it wasn't, nothing can stop attackers from using 
special tools.

I'm not saying this is large concern, yet this is a concern one 
should be aware of.

> IMHO I would still really like to see this patch applied 
> because:
> 
> 1. The patch is relatively small and matches the http and stream 
> modules
> 2. It makes the mail module consistent with the http and stream 
> modules which both support reuseport on their listen arguments
> 3. The current situation is clearly really bad, and other users 
> have reported that they're seeing the same issue. Your 
> suggestion is to recompile nginx with a particular option 
> disabled, but this isn't required for stream or http handlers, 
> just adding reuseport is an acceptable option to fix those 
> handlers, and I think it should be something mail handlers can 
> do as well.

The reuseport is implemented in nginx to mitigate lock contention 
issues on multiprocessor configurations in case of very high 
accept rates.  It is not available in mail because high accept 
rates are not expected to be the case in mail protocols.

While reuseport can be misused as a balancing solution, this is not 
something I can recommend, given various reuseport limitations and 
issues.  Notably, reuseport requires a lot of additional sockets, 
which is known to be a problem in some configurations, does not 
prevent use of the same address in other processes, which requires 
great care when managing running server, and opens a DoS vector to 
overload a particular worker process, as outlined above.

In the particular case you and others are reporting a better 
solution would be to address the issue which causes bad balancing.  
For now it looks like removing EPOLLEXCLUSIVE should fix things, 
making misuse of reuseport unnecessary in all modules.  Testing on 
real servers affected by the issue is appreciated.

-- 
Maxim Dounin
http://mdounin.ru/