[PATCH] SO_REUSEPORT support for listen sockets

[Maxim Dounin]

Hello!

On Sun, Jul 28, 2013 at 09:11:26PM +0800, Sepherosa Ziehau wrote:

[...]

> > 2) this feature should be disabled on DragonFly versions prior to the
> > 740d1d9 commit, because it clearly wouldn't do any good there,
> 
> On DragonFlyBSD, I could use a sysctl node to detect this feature.
> However, this obviously is OS specific, I am not quite sure about
> where to put that code.  Any hint on this?

DragonFly is currently handled as a variant of FreeBSD (see 
auto/os/conf, src/core/ngx_config.h, src/os/unix/ngx_freebsd_config.h, 
src/os/unix/ngx_freebsd_init.c).

If you want to do sysctl runtime checks, correct place would be in 
src/os/unix/ngx_freebsd_init.c, ngx_os_specific_init().  I'm not 
sure it worth the effort though, probably just assuming user knows 
better will be ok.

[...]

> > 4) how does this (OS-level sharding) play with nginx's upgrade process
> > (forking of new binary and passing listening fds)? Are there any
> > side-effects of this change that could result in dropped packets /
> > requests?
> 
> I am not quite sure about how nginx handles upgrade.  But if
> so_reuseport is enabled and worker process exits (assuming new worker
> is not forked by old worker), any pending sockets on listen socket's
> completion queue but not yet accept(2)'ed will be dropped (at least
> this is the case in DragonFlyBSD).

By "pending sockets completion queue" you mean exactly one 
socket's queue, as created by a worker process?  I.e., there is no 
mechanism to pass unaccepted connections from one socket to other 
sockets listening on the same address if the socket is closed 
(e.g. due to process exit)?

This sounds bad, as it will result in connections being lost not 
only on binary upgrades but also on normal configuration reloads.

Just for reference, upgrade process works as follows:

- old master fork()'s and then uses execve() to start a new master 
  process, with listenings sockets descriptors enumerated in the 
  environment

- new master process parses configuration (using inherited 
  listening sockets fds it got from old master) and fork()'s new 
  worker processes

- old master asks old worker processes to exit gracefully (to stop 
  accepting new connections and to exit once processing of 
  currently active requests is complete)

Configuration reload works as follows:

- master process loads new configuration

- master process fork()'s new worker processes (with a new configuration)

- master proces asks old worker processes to exit gracefully

Some documentation is here:

http://nginx.org/en/docs/control.html

-- 
Maxim Dounin
http://nginx.org/en/donation.html