high Traffic setup problem, module status don't deliver data

[Maxim Dounin]

Hello!

On Tue, Feb 11, 2014 at 02:14:14PM +0100, Aleksandar Lazic wrote:

> Hi.
> 
> Am 11-02-2014 13:28, schrieb Maxim Dounin:
> >Hello!
> >
> >On Tue, Feb 11, 2014 at 01:10:59PM +0100, Aleksandar Lazic wrote:
> >
> >>
> >>Am 11-02-2014 12:48, schrieb Maxim Dounin:
> >>>Hello!
> >>>
> >>>On Tue, Feb 11, 2014 at 12:34:59PM +0100, Aleksandar Lazic wrote:
> >>>
> >>
> >>[snipp]
> >>
> >>>>>You haven't shown your server level configuration.
> >>>>>Do you use deferred accept?
> >>>>
> >>>>yes
> >>>>
> >>>>listen       <IP>:80 deferred default_server;
> >>>
> >>>Try switching it off, there could be a problem if kernel decides
> >>>to switch to syncookies, see this ticket for details:
> >>>
> >>>http://trac.nginx.org/nginx/ticket/353
> >>>
> >>>(The problem is fixed in 1.5.10, and 1.4.5 will have the fix,
> >>>too.)
> >>
> >>Ok thanks.
> >>I have now removed deferred  and added backlog=1024
> >
> >Does it actually solve the problem?  It also would be intresting
> >to know what exactly did it - removing deferred or adding backlog?
> 
> Due to the fact that we have mostly in the morning our highest traffic I can
> tell you this tomorrow.
> I still search for a useable load testing setup.
> Currently I have one server (1 GB) with
> 
> https://github.com/wg/wrk
> 
> I will try today
> 
> https://bitbucket.org/yarosla/httpress/wiki/Home
> https://code.google.com/p/httperf/
> 
> But if anybody have a suggestion for distributed load testing service I'm
> open eared.

I personally prefer http_load, mostly for historical reasons.  We 
also use wrk here, which is quite good as well.

> What I also have seen is that.
> 
> netstat -s|egrep 'listen queue|SYNs to LISTEN sockets dropped'
>     131753 times the listen queue of a socket overflowed
>     18195732 SYNs to LISTEN sockets dropped
> 
> a second later.
> 
> netstat -s|egrep 'listen queue|SYNs to LISTEN sockets dropped'
>     131753 times the listen queue of a socket overflowed
>     18195743 SYNs to LISTEN sockets dropped
> 
> What could be this the reason?

I'm not really a Linux expert as I prefer FreeBSD, but number 
suggests there are no listen queue overflows during the second, 
but there are other SYN drops.  Looking into the code - there are lots 
of possible reasons, including various allocation failures and/or 
other edge cases.  It needs additional investigation to tell what 
goes on.

> >(As for backlog size, I usually set it to something big enough to
> >accomodate about 1 or 2 seconds of expected peek connection rate.
> >That is, 1024 is good enough for about 500 connections per second.
> >But with deferred on Linux, it looks like deferred connection are
> >sitting in the same queue as normal ones, and this may drastically
> >change things.)
> 
> OK. That means with deferred I should double or divide the listening value?

With deferred, it looks like all (potential) deferred connection 
should be added to the value.  And it's very hard to tell how many 
there will be.

-- 
Maxim Dounin
http://nginx.org/