Nginx TCP Delays

[Khalid Shaikh]

Maxim,
You may be a genius.

I see the backlog is only 1024, so I upped that
echo 10024 > /proc/sys/net/ipv4/tcp_max_syn_backlog

Things feel responsive, but I will only know more in 24 hours when we hit
peak loads.

Best,
Khalid

On Mon, Sep 21, 2009 at 9:48 AM, Maxim Dounin <mdounin at mdounin.ru> wrote:

> Hello!
>
> On Mon, Sep 21, 2009 at 09:26:22AM -0700, Khalid Shaikh wrote:
>
> > Team,
> > Here is another sample point.  See the 9 second delay on the server I
> keep
> > speaking about.  This problem shows across multiple servers.  The one we
> are
> > investigating is http://67.159.60.59/status.  Same nginx.conf across
> all.
> >
> > root at pa:~# clear; ./status.rb
> > Active connections: 17442
> > server accepts handled requests
> >  291457 291457 499538
> > Reading: 242 Writing: 3838 Waiting: 13362
> > http://208.53.158.42/status     0.07 seconds
> >
> > Active connections: 11545
> > server accepts handled requests
> >  8720635 8720635 13904180
> > Reading: 154 Writing: 1370 Waiting: 10021
> > http://67.159.60.37/status      0.03 seconds
> >
> > Active connections: 3807
> > server accepts handled requests
> >  3758087 3758087 27448407
> > Reading: 44 Writing: 213 Waiting: 3550
> > http://66.90.101.97/status      0.04 seconds
> >
> > Active connections: 8855
> > server accepts handled requests
> >  6865763 6865763 10978021
> > Reading: 133 Writing: 988 Waiting: 7734
> > http://208.53.158.75/status     0.03 seconds
> >
> > Active connections: 10571
> > server accepts handled requests
> >  194174 194174 467424
> > Reading: 106 Writing: 2147 Waiting: 8318
> > http://67.159.60.59/status      9.03 seconds
>
> After looking closely on timings you provide - it's
> suspiciously rounded delays.  Looks like packet
> drops/retransmissions.  This may be due to some network resource
> shortage on the host in question or some problems on the wire.
>
> First of all, try tcpdump to see what actually happens on the
> wire.
>
> Also it's probably a good idea to examine your backlog size and
> status.  AFAIK Linux by default just drops incoming SYN packets on
> listen queue overflows, so this may be the reason.
>
> Maxim Dounin
>
> >
> > Active connections: 8354
> > server accepts handled requests
> >  4314517 4314517 6679147
> > Reading: 195 Writing: 1954 Waiting: 6205
> > http://208.53.158.200/status    0.03 seconds
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://nginx.org/pipermail/nginx/attachments/20090921/74622757/attachment.html>