nginx imap proxy - timeouts

[Dominic]

Dear Falko

On 23/10/14 09:06, Falko Koenig wrote:
> Hello,
> 
> we are using nginx as imap and pop3 proxy with a ldap database querying
> the correct destination server for the user. LDAP Requests are realized
> by a perl script for using different LDAP servers. We have noticed that
> nginx has temporarily timeouts querying the LDAP server. The timeouts
> occur before the query is sent to the LDAP server by the nginx. We get
> the following error code:
> 
> 2014/10/22 17:16:49 [error] 2073#0: *4391695 auth http server
> 127.0.0.1:8000 timed out (110: Connection timed out) while in http auth
> state, client: XXX.XXX.XXX.XXX, server: 0.0.0.0:993, login: "USERNAME"
> 2014/10/22 17:16:52 [error] 2073#0: *4393196 auth http server
> 127.0.0.1:8000 timed out (110: Connection timed out) while in http auth
> state, client: XXX.XXX.XXX.XXX, server: 0.0.0.0:995, login: "USERNAME"
> 
> Because of that it isn't possible that the answer of the LDAP server is
> sent too slowly back to the nginx. The servers are in a cluster
> configuration and one server processes 20.000 Connections. Are there any
> required kernel parameters for solving the problem?

My random guess.

Linux has a maximum of open connections.
You can get the number by the following command:

sysctl net.ipv4.ip_local_port_range
> net.ipv4.ip_local_port_range = 32768    61000

The default on my host are 28232 local ports.  For each connection a
local port is used. Because IMAP/POP3 are persistent connections, the
local ports are getting rare on your system.
You can increase this setting by /etc/sysctl.conf [1][2].

I guess your Perl script is a small CGI server running on localhost on
port XYZ (something like port 9000). This will add up additional local
port usages for every LDAP lookup. If you have 20'000 current
connection, 8000 used ports that are not yet been freed by the network
stack (see notes in [1]) and 2000 concurrent LDAP Perl request you'll
reach the default limit of 28232 possible ports.
If your Perl script running on localhost I would recommend switching to
a Unix socket to save up additional local ports.

The Linux network stack (=> limitation of the IPv4 protocol) is not
capable of handling more than 65536 open connections [3].
An alternative to this problem might be a switch of a *BSD operating
system. Whatsapp is handling more than 2 million connections per host
with FreeBSD [4]. Or use a load balancer with multiple nodes.

Regards
Dominic


[1] http://www.nateware.com/linux-network-tuning-for-2013.html
[2] http://dak1n1.com/blog/12-nginx-performance-tuning
[3]
http://superuser.com/questions/251596/is-there-a-hard-limit-of-65536-open-tcp-connections-per-ip-address-on-linux
[4] http://blog.whatsapp.com/196/1-million-is-so-2011