I like your analysis of the SYN packets. The tests are specifically localhost, so it cannot be network for those connections.<br><br>I will investigate based on your analysis.<br><br>Best,<br>Khalid<br><br><div class="gmail_quote">
On Mon, Sep 21, 2009 at 9:48 AM, Maxim Dounin <span dir="ltr"><<a href="mailto:mdounin@mdounin.ru">mdounin@mdounin.ru</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hello!<br>
<div><div></div><div class="h5"><br>
On Mon, Sep 21, 2009 at 09:26:22AM -0700, Khalid Shaikh wrote:<br>
<br>
> Team,<br>
> Here is another sample point. See the 9 second delay on the server I keep<br>
> speaking about. This problem shows across multiple servers. The one we are<br>
> investigating is <a href="http://67.159.60.59/status" target="_blank">http://67.159.60.59/status</a>. Same nginx.conf across all.<br>
><br>
> root@pa:~# clear; ./status.rb<br>
> Active connections: 17442<br>
> server accepts handled requests<br>
> 291457 291457 499538<br>
> Reading: 242 Writing: 3838 Waiting: 13362<br>
> <a href="http://208.53.158.42/status" target="_blank">http://208.53.158.42/status</a> 0.07 seconds<br>
><br>
> Active connections: 11545<br>
> server accepts handled requests<br>
> 8720635 8720635 13904180<br>
> Reading: 154 Writing: 1370 Waiting: 10021<br>
> <a href="http://67.159.60.37/status" target="_blank">http://67.159.60.37/status</a> 0.03 seconds<br>
><br>
> Active connections: 3807<br>
> server accepts handled requests<br>
> 3758087 3758087 27448407<br>
> Reading: 44 Writing: 213 Waiting: 3550<br>
> <a href="http://66.90.101.97/status" target="_blank">http://66.90.101.97/status</a> 0.04 seconds<br>
><br>
> Active connections: 8855<br>
> server accepts handled requests<br>
> 6865763 6865763 10978021<br>
> Reading: 133 Writing: 988 Waiting: 7734<br>
> <a href="http://208.53.158.75/status" target="_blank">http://208.53.158.75/status</a> 0.03 seconds<br>
><br>
> Active connections: 10571<br>
> server accepts handled requests<br>
> 194174 194174 467424<br>
> Reading: 106 Writing: 2147 Waiting: 8318<br>
> <a href="http://67.159.60.59/status" target="_blank">http://67.159.60.59/status</a> 9.03 seconds<br>
<br>
</div></div>After looking closely on timings you provide - it's<br>
suspiciously rounded delays. Looks like packet<br>
drops/retransmissions. This may be due to some network resource<br>
shortage on the host in question or some problems on the wire.<br>
<br>
First of all, try tcpdump to see what actually happens on the<br>
wire.<br>
<br>
Also it's probably a good idea to examine your backlog size and<br>
status. AFAIK Linux by default just drops incoming SYN packets on<br>
listen queue overflows, so this may be the reason.<br>
<font color="#888888"><br>
Maxim Dounin<br>
</font><div><div></div><div class="h5"><br>
><br>
> Active connections: 8354<br>
> server accepts handled requests<br>
> 4314517 4314517 6679147<br>
> Reading: 195 Writing: 1954 Waiting: 6205<br>
> <a href="http://208.53.158.200/status" target="_blank">http://208.53.158.200/status</a> 0.03 seconds<br>
<br>
</div></div></blockquote></div><br>