[PATCH] SO_REUSEPORT support for listen sockets

[Sepherosa Ziehau]

On Fri, Jul 26, 2013 at 6:59 PM, Piotr Sikora <piotr at cloudflare.com> wrote:
> Hey,
>
>> @@ -76,6 +78,13 @@
>>       0,
>>       NULL },
>>
>> +    { ngx_string("so_reuseport"),
>> +      NGX_MAIN_CONF|NGX_DIRECT_CONF|NGX_CONF_TAKE1,
>> +      ngx_set_so_reuseport,
>> +      0,
>> +      0,
>> +      NULL },
>> +
>>     { ngx_string("debug_points"),
>>       NGX_MAIN_CONF|NGX_DIRECT_CONF|NGX_CONF_TAKE1,
>>       ngx_conf_set_enum_slot,
>> @@ -1361,3 +1370,24 @@
>>
>>     return NGX_CONF_OK;
>> }
>> +
>> +
>> +static char *
>> +ngx_set_so_reuseport(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
>> +{
>> +    ngx_str_t        *value;
>> +    ngx_core_conf_t  *ccf;
>> +
>> +    ccf = (ngx_core_conf_t *) conf;
>> +
>> +    value = (ngx_str_t *) cf->args->elts;
>> +
>> +    if (ngx_strcmp(value[1].data, "on") == 0) {
>> +        ccf->so_reuseport = 1;
>> +    } else if (ngx_strcmp(value[1].data, "off") == 0) {
>> +        ccf->so_reuseport = 0;
>> +    } else {
>> +        return "invalid value";
>> +    }
>> +    return NGX_CONF_OK;
>> +}
>
> This can be replaced with ngx_conf_set_flag_slot(), i.e.:
>
> +    { ngx_string("so_reuseport"),
> +      NGX_MAIN_CONF|NGX_DIRECT_CONF|NGX_CONF_FLAG,
> +      ngx_conf_set_flag_slot,
> +      0,
> +      offsetof(ngx_core_conf_t, so_reuseport),
> +      NULL },
>
> Also:
> 1) like Tom said, you definitely need to guard code to make sure
> SO_REUSEPORT is available,

Yeah.  I will take care of it.

> 2) this feature should be disabled on DragonFly versions prior to the
> 740d1d9 commit, because it clearly wouldn't do any good there,

On DragonFlyBSD, I could use a sysctl node to detect this feature.
However, this obviously is OS specific, I am not quite sure about
where to put that code.  Any hint on this?

> 3) it might make sense to expose this as an option of "listen"
> directive, instead of a global setting,

I tried to do so_reuseport as "listen" option.  However, the accept
mutex to be disabled is actually global, as far as I understand the
code.  That's why I currently implement so_reuseport as global option
instead of "listen" option.

> 4) how does this (OS-level sharding) play with nginx's upgrade process
> (forking of new binary and passing listening fds)? Are there any
> side-effects of this change that could result in dropped packets /
> requests?

I am not quite sure about how nginx handles upgrade.  But if
so_reuseport is enabled and worker process exits (assuming new worker
is not forked by old worker), any pending sockets on listen socket's
completion queue but not yet accept(2)'ed will be dropped (at least
this is the case in DragonFlyBSD).

Best Regards,
sephe

-- 
Tomorrow Will Never Die