[Patch] SO_REUSEPORT support from master process

[Lu, Yingqi]

Hi Sephe,

Thanks for your email.

The main difference between your patch and this prototype is that we create certain number of duplicated listen sockets in the master process instead of child processes. All the child processes have access to the same set of the duplicated listen sockets. In this case, all the child processes are the same, there is no special child needed. I think this addresses the Nginx folks concern.

Also, the number of the duplicated listen sockets is calculated as 1/8 of the number of active CPU threads (when the system has only 8 or less active threads, there is no duplication). It has nothing to do with the number of the child process. We use this way to extend the system CPU scalability (when needed) as well as make sure there will not be a lot of listen sockets open in the kernel.

Our prototype also testes the availability of the SO_REUSEPORT feature. When the feature is not available, it will fall back to the default behavior. I test this on Linux Kernel 3.8.8 where the feature is not available.

Thanks,
Yingqi

-----Original Message-----
From: nginx-devel-bounces at nginx.org [mailto:nginx-devel-bounces at nginx.org] On Behalf Of Sepherosa Ziehau
Sent: Wednesday, August 27, 2014 2:09 AM
To: nginx-devel at nginx.org
Subject: Re: [Patch] SO_REUSEPORT support from master process

Does Linux handle un'accepted(2)' sockets inheritance between SO_REUSEPORT sockets, if one of the SO_REUSEPORT socket is closed?
It's one of the concern that nginx folks have about SO_REUSEPORT.  I am not following Linux, so I am not sure about it at all.  That's also why I wanted to keep one SO_REUSEPORT socket opened.

BTW, DragonFly had my patch enabled by default in the dports system for a long time.

And many systems have the symbol of SO_REUSEPORT and the setsockopt works too, e.g. FreeBSD and probably other BSDs, but on kernel side, SO_REUSEPORT does not work in the fashion like Linux or DragonFly.  So even simple run-time test would not be able detect the "new"
SO_REUSEPORT support.

Best Regards,
sephe


On Sat, Aug 23, 2014 at 12:55 AM, Lu, Yingqi <yingqi.lu at intel.com> wrote:
> Dear All,
>
>
>
> The “SO_REUSEPORT support for listen sockets support” patches 
> submitted by Sepherosa Ziehau are posted and discussed in [1] and [2]. 
> Last update on the threads was 09/05/2013 and the patch is not 
> included in the current Nginx code. Reading from the discussion, my 
> understanding is that his patch makes a dedicated listen socket for 
> each of the child process. In order to make sure at any given time 
> there is always a listen socket available, the patch makes the first worker process different/special than the rest.
>
>
>
> Here, I am proposing a simpler way to enable the SO_REUSEPORT support. 
> It is just to create and configure certain number of listen sockets in 
> the master process with SO_REUSEPORT enabled. All the children processes can inherit.
> In this case, we do not need to worry about ensuring 1 available 
> listen socket at the run time. The number of the listen sockets to be 
> created is calculated based on the number of active CPU threads. With 
> big system that has more CPU threads (where we have the scalability 
> issue), there are more duplicated listen sockets created to improve the throughput and scalability.
> With system that has only 8 or less CPU threads, there will be only 1 
> listen socket. This makes sure duplicated listen sockets only being 
> created when necessary. In case that SO_REUSEPORT is not supported by 
> the OS, it will fall back to the default/original behavior (this is 
> tested on Linux kernel
> 3.8.8 where SO_REUSEPORT is not supported).
>
>
>
> This prototype patch has been tested on an Intel modern dual socket 
> platform with a three tier open source web server workload 
> (PHP+Nginx/memcached/MySQL). The web server has 2 IP network 
> interfaces configured for testing. The Linux kernel used for testing 
> is 3.13.9. Data
> show:
>
>
>
> Case 1: with single listen statement (Listen 80) specified in the 
> configuration file, there is 46.3% throughout increase.
>
> Case 2: with dual listen statements (for example, Listen 
> 192.168.1.1:80 and Listen 192.168.1.2:80), there is 10% throughput increase.
>
>
>
> Both testing cases keep everything the same except the patch itself to 
> get above result.
>
>
>
> The reason that Case1 has bigger performance gains is that Case1 by 
> default only has 1 listen socket while Case2 by default already has 2.
>
>
>
> Please review it and let me know your questions and comments.
>
>
>
> [1] http://forum.nginx.org/read.php?29,241283,241283
>
> [2] http://forum.nginx.org/read.php?29,241470,241470
>
>
>
> 1. Software and workloads used in performance tests may have been 
> optimized for performance only on Intel microprocessors. Performance 
> tests, such as SYSmark and MobileMark, are measured using specific 
> computer systems, components, software, operations and functions. Any 
> change to any of those factors may cause the results to vary. You 
> should consult other information and performance tests to assist you 
> in fully evaluating your contemplated purchases, including the 
> performance of that product when combined with other products.
>
>
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel



--
Tomorrow Will Never Die

_______________________________________________
nginx-devel mailing list
nginx-devel at nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel