Strange $upstream_response_time latency spikes with reverse proxy

[Jay Oster]

Hi Maxim,

On Tue, Mar 19, 2013 at 7:19 AM, Maxim Dounin <mdounin at mdounin.ru> wrote:

> Hello!
>
> As far as I understand, tcp_max_syn_backlog configures global
> cumulative limit for all listening sockets, while somaxconn limits
> one listening socket backlog.  If any of the two is too small -
> you'll see SYN packets dropped.
>
> > > Make sure to check actual listen queue sizes used on listen
> > > sockets involved.  On Linux (you are using Linux, right?) this
> > > should be possible with "ss -nlt" (or "netstat -nlt").
> >
> >
> > According to `ss -nlt`, send-q on these ports is set to 128. And recv-q
> on
> > all ports is 0. I don't know what this means for recv-q, use default? And
> > would default be 1024?
>
> In "ss -nlt" output send-q column is used to display listen queue
> size for listen sockets.  Number 128 here means you have listen
> queue for 128 connections only.  You should tune your backends to
> use bigger listen queues, 128 is certanly too small for concurency
> 5000 you use in your tests.
>
> (The recv-q column should indicate current number of connections
> in listen queue.)


This is an excellent tip, thank you! Regardless of whether it fully
resolves this issue, I will see about tuning the individual listen socket
queues. The server is using libevent's asynchronous HTTP server module, so
I'm not sure how much control I have over the socket options. But I will
investigate.


> > But according to `netstat -nlt` both queues are 0?
>
> This means that netstat isn't showing listen queue sizes on your
> host.  It looks like many linux systems still always display 0 for
> listen sockets.


Pretty strange. Oh well. `ss` works for me.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20130319/18cc8594/attachment.html>