Nginx TCP Delays

[Maxim Dounin]

Hello!

On Mon, Sep 21, 2009 at 09:26:22AM -0700, Khalid Shaikh wrote:

> Team,
> Here is another sample point.  See the 9 second delay on the server I keep
> speaking about.  This problem shows across multiple servers.  The one we are
> investigating is http://67.159.60.59/status.  Same nginx.conf across all.
> 
> root at pa:~# clear; ./status.rb
> Active connections: 17442
> server accepts handled requests
>  291457 291457 499538
> Reading: 242 Writing: 3838 Waiting: 13362
> http://208.53.158.42/status     0.07 seconds
> 
> Active connections: 11545
> server accepts handled requests
>  8720635 8720635 13904180
> Reading: 154 Writing: 1370 Waiting: 10021
> http://67.159.60.37/status      0.03 seconds
> 
> Active connections: 3807
> server accepts handled requests
>  3758087 3758087 27448407
> Reading: 44 Writing: 213 Waiting: 3550
> http://66.90.101.97/status      0.04 seconds
> 
> Active connections: 8855
> server accepts handled requests
>  6865763 6865763 10978021
> Reading: 133 Writing: 988 Waiting: 7734
> http://208.53.158.75/status     0.03 seconds
> 
> Active connections: 10571
> server accepts handled requests
>  194174 194174 467424
> Reading: 106 Writing: 2147 Waiting: 8318
> http://67.159.60.59/status      9.03 seconds

After looking closely on timings you provide - it's 
suspiciously rounded delays.  Looks like packet 
drops/retransmissions.  This may be due to some network resource 
shortage on the host in question or some problems on the wire.

First of all, try tcpdump to see what actually happens on the 
wire.

Also it's probably a good idea to examine your backlog size and 
status.  AFAIK Linux by default just drops incoming SYN packets on 
listen queue overflows, so this may be the reason.

Maxim Dounin

> 
> Active connections: 8354
> server accepts handled requests
>  4314517 4314517 6679147
> Reading: 195 Writing: 1954 Waiting: 6205
> http://208.53.158.200/status    0.03 seconds