[Patch] SO_REUSEPORT support from master process

[Sepherosa Ziehau]

On Thu, Aug 28, 2014 at 12:23 AM, Lu, Yingqi <yingqi.lu at intel.com> wrote:
> Hi Sephe,
>
> Thanks for your email.
>
> The main difference between your patch and this prototype is that we create certain number of duplicated listen sockets in the master process instead of child processes. All the child processes have access to the same set of the duplicated listen sockets. In this case, all the child processes are the same, there is no special child needed. I think this addresses the Nginx folks concern.
>


As I have said, the special last listen socket is for inheriting other
listen sockets '3-way hand shaken but not accepted' sockets.
It is used to support non-disruptive binary upgrading.  Kernel side
also has work to do, that's why I asked about how linux handles
non-disruptive binary upgrading in kernel.  e.g. in DragonFly:
S1: s1 s2 s3
S2: s4 s5 s6
If S2 is closed, then
S1: s1 s2 s3 s4 s5 s6

S1 and S2 are SO_REUSEPORT listen sockets, s1~s6 are '3-way hand
shaken but not accepted' sockets.

Maybe you should check Linux's SO_REUSEPORT implementation first.


> Also, the number of the duplicated listen sockets is calculated as 1/8 of the number of active CPU threads (when the system has only 8 or less active threads, there is no duplication). It has nothing to do with the number of the child process. We use this way to extend the system CPU scalability (when needed) as well as make sure there will not be a lot of listen sockets open in the kernel.
>


It's OS specific, in DragonFly, we get best performance if worker
process uses its own listen socket due to the nature of its network
stack design.


> Our prototype also testes the availability of the SO_REUSEPORT feature. When the feature is not available, it will fall back to the default behavior. I test this on Linux Kernel 3.8.8 where the feature is not available.


As I said, SO_REUSEPORT is defined even in BSD 4.4.  And
setsockopt(s, SOL_SOCKET, SO_REUSEPORT, &on, sizeof(on));
will return 0 on all BSDs.  But FreeBSD and other BSDs SO_REUSEPORT
obviously do _not_ work in the way like Linux and DragonFly.

Best Regards,
sephe