[PATCH] SO_REUSEPORT support for listen sockets

[Sepherosa Ziehau]

On Mon, Jul 29, 2013 at 10:57 PM, Maxim Dounin <mdounin at mdounin.ru> wrote:
> Hello!
>
> On Sun, Jul 28, 2013 at 09:11:26PM +0800, Sepherosa Ziehau wrote:
>
> [...]
>
>> > 2) this feature should be disabled on DragonFly versions prior to the
>> > 740d1d9 commit, because it clearly wouldn't do any good there,
>>
>> On DragonFlyBSD, I could use a sysctl node to detect this feature.
>> However, this obviously is OS specific, I am not quite sure about
>> where to put that code.  Any hint on this?
>
> DragonFly is currently handled as a variant of FreeBSD (see
> auto/os/conf, src/core/ngx_config.h, src/os/unix/ngx_freebsd_config.h,
> src/os/unix/ngx_freebsd_init.c).
>
> If you want to do sysctl runtime checks, correct place would be in
> src/os/unix/ngx_freebsd_init.c, ngx_os_specific_init().  I'm not
> sure it worth the effort though, probably just assuming user knows
> better will be ok.

Agree.  We will just trust users.

>> I am not quite sure about how nginx handles upgrade.  But if
>> so_reuseport is enabled and worker process exits (assuming new worker
>> is not forked by old worker), any pending sockets on listen socket's
>> completion queue but not yet accept(2)'ed will be dropped (at least
>> this is the case in DragonFlyBSD).
>
> By "pending sockets completion queue" you mean exactly one
> socket's queue, as created by a worker process?  I.e., there is no
> mechanism to pass unaccepted connections from one socket to other
> sockets listening on the same address if the socket is closed
> (e.g. due to process exit)?
>
> This sounds bad, as it will result in connections being lost not
> only on binary upgrades but also on normal configuration reloads.
>
> Just for reference, upgrade process works as follows:
>
> - old master fork()'s and then uses execve() to start a new master
>   process, with listenings sockets descriptors enumerated in the
>   environment
>
> - new master process parses configuration (using inherited
>   listening sockets fds it got from old master) and fork()'s new
>   worker processes
>
> - old master asks old worker processes to exit gracefully (to stop
>   accepting new connections and to exit once processing of
>   currently active requests is complete)
>
> Configuration reload works as follows:
>
> - master process loads new configuration
>
> - master process fork()'s new worker processes (with a new configuration)
>
> - master proces asks old worker processes to exit gracefully
>
> Some documentation is here:
>
> http://nginx.org/en/docs/control.html

Thank you very much for the hint!  The patch needs some changes to
handle this, as well as DragonFly's kernel.

After I have done the DragonFly kernel part, I would do the following
changes to the current patch:
If so_reuseport is enabled, the master will open the listen sockets
and after forking the first worker, the master closes all opened
listen sockets (they are inherited by the first worker); so the kernel
would have at least one listen socket to migrate the
completed-but-not-yet-accepted sockets of the to-be-closed listen
sockets.

Best Regards,
sephe

-- 
Tomorrow Will Never Die