Very odd epoll_ctl calls in mail proxy

[Igor Sysoev]

On Wed, Nov 09, 2011 at 02:47:21PM +0400, Igor Sysoev wrote:
> On Wed, Nov 09, 2011 at 09:54:32AM +1100, Robert Mueller wrote:
> > 
> > > It seems you have about 170 listen directives.
> > 
> > Indeed we do! Add together ipv4 + ipv6 + a failover ips + a number of
> > alternative ports and configurations, and it does add up!
> > 
> > > This is probably caused by accept_mutex. Try
> > > 
> > > events {
> > >      accept_mutex  off;
> > >      ...
> > > }
> > > 
> > > Probably it's better to disable accept_mutex by default.
> > 
> > Ah right, makes sense, I've gone and added that.
> > 
> > Not sure if it's an issue, but an alternative would be to create a
> > separate "accepting" fd, add all the listening fd's to that one, and
> > only add/remove that fd from the main polling set when you have the
> > mutex.
> > 
> > That at least should work for epoll, which says:
> > 
> > Q3  Is the epoll file descriptor itself poll/epoll/selectable?
> > A3  Yes.  If an epoll file descriptor has events waiting then it will
> > indicate as being readable.
> 
> Yes, this method should work with epoll and kqueue, but not sure about
> Solaris /dev/poll and event ports.
> 
> > Not sure if it's worth it though, given you can just turn off the accept
> > mutex, and apparently linux no longer has a thundering herd issue:
> > 
> > http://stackoverflow.com/questions/2213779/does-the-thundering-herd-problem-exist-on-linux-anymore
> > 
> > I'd like to see an LKML post about that though...
> 
> Could you tell is any difference in CPU usage or in "top" if accept_mutex
> is off in your system ? How much accepts per second does nginx process ?

I've just tested on FreeBSD 8.2-STABLE/amd64:

FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 SMT threads

8 workers eat ~1% more without accept_mutex on ~240 accepts/second.


-- 
Igor Sysoev