[patch] Set SO_REUSEADDR on outgoing TCP connections

[Marek Majkowski]

Testing the patch is straightforward once you have an idea how to do it.

1) Reduce ephemeral port range

$ sudo -s bash -c "echo 32000 32004 > /proc/sys/net/ipv4/ip_local_port_range"

2) Start nginx with simple config from this gist.

https://gist.github.com/anonymous/10285909

3) The gist also contains two simple python scripts. They establish
connections to local nginx instance, and _don't_ use ephemeral port
range, therefore not interfere with the mechanisms described in this patch.

Run the scripts against patched and unpatched nginx. The first one
connects to multiple destinations:

$ python connect_multi.py

You should see that patched nginx can indeed establish more connection
before running out of ports.

$ python connect_single.py

You should see that both patched and unpatched nginx are able to open
exactly the same number of connections to a single destination.

4) Looking at outgoing SYN's and FIN's will help debugging:

$ sudo tcpdump -ni any 'port 80 and ((tcp[tcpflags] & tcp-syn != 0 and
tcp[tcpflags] & tcp-ack == 0) or (tcp[tcpflags] & tcp-fin != 0))'

5) Running nginx with strace could also give some hints
strace -f -e trace=bind,connect,close,setsockopt,socket ./objs/nginx ....

Marek

On Wed, Apr 9, 2014 at 4:53 PM, Marek Majkowski <majek04 at gmail.com> wrote:
> Usually, when establishing a connection the kernel allocates outgoing
> TCP/IP port automatically from an ephemeral port range. Unfortunately
> when selecting the outgoing source IP (using bind before connect) the
> kernel needs a unique port number. As the result it can only establish
> a single outgoing connection from a single source port. This can cause
> problems with a large number of outgoing proxy connections - it's
> possible for the kernel to run out free ports in the ephemeral range.