[PATCH] SO_REUSEPORT support for listen sockets

[Sepherosa Ziehau]

Hi all,

I have the following preliminary patch to enable SO_REUSEPORT feature
on listen sockets:
http://leaf.dragonflybsd.org/~sephe/ngx_soreuseport.diff

The basic idea of the above patch is:
- Defer the listen socket creation until work processes are forked
- Work process creates listen socket, and set SO_REUSEPORT before bind(2)
- Accept mutex is no longer needed, since worker process is not
contended on the single listen socket anymore


The SO_REUSEPORT sockopt on Linux:
https://lwn.net/Articles/542629/

The SO_REUSEPORT sockopt on DragonFlyBSD:
http://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/740d1d9f7b7bf9c9c021abb8197718d7a2d441c9

The non-blocking accept(2) w/ SO_REUSEPORT performance improvement on
DragonFlyBSD:
http://lists.dragonflybsd.org/pipermail/users/2013-July/053632.html


The preliminary httperf test shows w/ "so_reuseport on" gives me ~33%
req/s performance improvement on DragonFlyBSD:

httperf is running as:
httperf --server=$server_name --wsess=5000,1,1 --max-conn=4

Same testing machines and network configuration as in:
http://lists.dragonflybsd.org/pipermail/users/2013-July/053632.html

Each client runs 16 above httperf test, except the box w/ bce, which
runs 8 above httperf.

The nginx w/ "so_reuseport on" is doing 49852 reqs/s (4 run avg) and
there are 35%~40% idle time on each hyperthread.
The nginx w/o "so_reuseport on" is doing 37386 reqs/s (4 run avg).


Any feedbacks are welcome.

Best Regards,
sephe

-- 
Tomorrow Will Never Die