[PATCH] Mail: add the "reuseport" option of the "listen" directive

[Michael Kliewe]

Am 18.08.2021 um 03:14 schrieb Robert Mueller:
>> First, thanks for the patch.
>>
>> While the reuseport could cure (or hide if you will) the unbalancing you
>> see it makes sense to get better understanding what exactly is going on.
>>   So far we haven't seen such weird behaviour ourself neither received
>> reports about such uneven connections distribution among nginx workers.
>>
>> Any chances you have accept_mutex and/or multi_accept?  Any other ideas?
> xxxx
>
> As you can see, without the reuseport option, this causes severe scalability problems for us.

Hi,

I just checked, we see the same problem, the connection distribution is 
very uneven. One process is using 30% CPU, one 15% CPU, and the rest is 
below 5%.

# for i in 1285 1286 1287 1288 1289 1290 1291 1292; do echo "$i - " `ls 
/proc/$i/fd | wc -l`; done
1285 -  106
1286 -  9447
1287 -  430
1288 -  2222
1289 -  76
1290 -  48
1291 -  24447
1292 -  42

We are using
nginx version: nginx/1.20.1

Configuration snippet:
```
worker_processes auto;
worker_rlimit_nofile 120000;

events {
     worker_connections 60000;
}

http {
...
}

mail {
   auth_http  127.0.0.1/mailauthXXXXX;

   proxy     on;
   starttls  on; ## enable STARTTLS for all mail servers

   ssl_prefer_server_ciphers  on;
   ssl_protocols              TLSv1.3 TLSv1.2 TLSv1.1 TLSv1;
   ssl_ciphers                xxxx;
   ssl_session_cache          shared:TLSSL:16m;
   ssl_session_timeout        10m;

   imap_auth login plain;
   ## STARTTLS is appended because of starttls directive above, 
AUTH=LOGIN and AUTH=PLAIN are also appended automatically
   imap_capabilities  "IMAP4rev1" "LITERAL+" "SASL-IR" "LOGIN-REFERRALS" 
"ID" "ENABLE" "IDLE" "NAMESPACE";
   pop3_capabilities  "TOP" "USER" "UIDL";

   server {
     listen                   a.b.c.d:993 ssl;
     listen                   [xxx:xxx:xxx:xxx::xxx]:993 ssl;
     protocol                 imap;
     server_name              imap.domain.com;
     auth_http_header         X-Auth-Port  993;
     auth_http_header         X-Domain     "imap.domain.com";
     auth_http_header         X-Auth       "xxxx";
     auth_http_header         User-Agent   "Nginx POP3/IMAP4 proxy";
     proxy_pass_error_message on;
     ssl_certificate          xxx.crt;
     ssl_certificate_key      xxx.key;
   }
}
```

Just wanted to show that you are not alone, Rob :-)

Michael