Strange $upstream_response_time latency spikes with reverse proxy

[Maxim Dounin]

Hello!

On Sun, Mar 17, 2013 at 02:23:20AM -0700, Jason Oster wrote:

[...]

> > 1) A trivial one.  Listen queue of your backend service is 
> > exhausted, and the SYN packet is dropped due to this.  This 
> > can be easily fixed by using bigger listen queue, and also 
> > easy enough to track as there are listen queue overflow 
> > counters available in most OSes.
> 
> Overflow queue is configured to 1024 on these hosts, though 
> nothing changes when I increase it. I can however make the delay 
> much longer by making the queue smaller.

On "these hosts"?  Note that listen queue aka backlog size is 
configured in _applications_ which call listen().  At a host level 
you may only configure somaxconn, which is maximum allowed listen 
queue size (but an application may still use anything lower, even 
just 1).

Make sure to check actual listen queue sizes used on listen 
sockets involved.  On Linux (you are using Linux, right?) this 
should be possible with "ss -nlt" (or "netstat -nlt").

> > 2) Some other queue in the network stack is exhausted.  This 
> > might be nontrivial to track (but usually possible too).
> 
> This is interesting, and could very well be it! Do you have any 
> suggestions on where to start looking?

I'm not a Linux expert, but quick search suggests it should be 
possible with dropwatch, see e.g. here:

http://prefetch.net/blog/index.php/2011/07/11/using-netstat-and-dropwatch-to-observe-packet-loss-on-linux-servers/

-- 
Maxim Dounin
http://nginx.org/en/donation.html