<div dir="ltr">Hi Will,<div><br></div><div><span style="font-size:12.8px">> * In the packet capture from the server, I see the SYN packet come in, then 3 more retransmits of that same syn come in before the server sent back the SYN/ACK. To me this indicates the issue in kernel or nginx side.</span><br></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Many times clients sends multiple SYN's. You </span><span style="font-size:12.8px">can run wireshark to check</span></div><div><span style="font-size:12.8px">the time stamps. If the packets are very close </span><span style="font-size:12.8px">(in milliseconds), that is normal,</span></div><div><span style="font-size:12.8px">else you have a problem on the server.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">nginx does not come into the picture during TCP handshake, it's job is</span></div><div><span style="font-size:12.8px">done when nginx indicates </span><span style="font-size:12.8px">that this socket is ready to accept connection</span></div><div><span style="font-size:12.8px">using the listen() system call. Once </span><span style="font-size:12.8px">the final </span><span style="font-size:12.8px">ack is done, the connection is</span></div><div><span style="font-size:12.8px">ready and if an accept() is called, it will </span><span style="font-size:12.8px">succeed (as in does-not-block).</span></div><div><span style="font-size:12.8px">However, the client would get success on </span><span style="font-size:12.8px">connect() at the time the TCP</span></div><div><span style="font-size:12.8px">handshake finished, not when the application finished the accept() call.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Maybe attaching tcpdump will be useful for someone to take a look at what</span></div><div><span style="font-size:12.8px">is wrong. Are the initial packets being dropped at the kernel due to bad</span></div><div><span style="font-size:12.8px">checksums? Do you have any IPTable rules that might drop syn's or rate limit</span></div><div><span style="font-size:12.8px">it? Do you see retransmissions (netstat -s)? Maybe you can run netstat -s</span></div><div><span style="font-size:12.8px">before and after to see which counters increase and derive some clues</span></div><div><span style="font-size:12.8px">from </span><span style="font-size:12.8px">that?</span></div><div><span style="font-size:12.8px"><br></span></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Sep 4, 2016 at 5:36 AM, Will Platnick <span dir="ltr"><<a href="mailto:wplatnick@gmail.com" target="_blank">wplatnick@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><span style="line-height:1.5">Hello,</span><br></div><div>I have run into a very interesting issue. I am replacing a set of nginx reverse proxy servers with a new set running an updated OS/nginx. These nginx servers front a popular API that's mostly used by mobile apps, but also a website that's hosted on a nearby subnet. I put the new servers into service last night, and this morning as traffic picked up (only a couple thousand requests per second), I got alerts from my DNS provider that requests to the new server were starting to timeout in the Connect phase. I hopped into New Relic, and I could see tons of requests from my website to the nginx reverse proxy timing out after it hit our limit of 10s. I did some curl requests with timing information, and I could see long times only in the time_connect level, confirming the issue was only in the connection phase. I hopped on the new nginx server and started a packet capture filtered to a machine on a nearby subnet, did the curl from there, got it taking a 9+ seconds in the connect phase, stopped the packet capture, and moved the traffic over to my old setup. No issues over there. </div><div><br></div><div>Here's everything I know/think is relevant:</div><div><br></div><div>* In the packet capture from the server, I see the SYN packet come in, then 3 more retransmits of that same syn come in before the server sent back the SYN/ACK. To me this indicates the issue in kernel or nginx side.</div><div><br></div><div>* There's absolutely no slowdown in the backends as measured from the same nginx server.</div><div><br></div><div>* There's nothing in the nginx error log </div><div><br></div><div>* There's nothing from the kernel in dmesg when this is happening</div><div><br></div><div>* NIC duplex is fine, no dropped queues from ethtool -S (but, again, it doesn't seem like a networking issue, we got the SYNs just fine, we just didn't send the syn/ack)</div><div><br></div><div>* I tried to artificially load test afterwords using ab and <a href="http://loader.io" target="_blank">loader.io</a>, doing 3x as many requests, but couldn't replicate the issue. I'm not sure if it's some weird issue due to misbehaving mobile clients and SSL filling up some sort of queue, but whatever it is, I can't replicate the issue on demand.</div><div><br></div><div>* Load on the box was fine (<4) and no crazy I/O.</div><div><br></div><div>* Keepalives were turned on</div><div><br></div><div>* Some relevant sysctl values:</div><div><br></div><div>cat /proc/sys/net/core/somaxconn (backlog is set to the same in the nginx config)</div><div>16384</div><div><br></div><div>cat /proc/sys/net/core/netdev_max_<wbr>backlog</div><div>15000</div><div> </div><div>cat /proc/sys/net/ipv4/tcp_max_<wbr>syn_backlog</div><div>262144</div><div><br></div><div>NGINX: 1.11.3</div><div>OS: Ubuntu 16.04.1 x64</div><div>Kernel: 4.4.0-36-generic</div><div><br></div><div>It seems to me the issue is at the kernel/app level, but I can't think of where to go from here. </div><div><br></div><div>If anybody has any ideas for me try, or if I've forgotten to mention something relevant, please let me know.</div></div>
<br>______________________________<wbr>_________________<br>
nginx mailing list<br>
<a href="mailto:nginx@nginx.org">nginx@nginx.org</a><br>
<a href="http://mailman.nginx.org/mailman/listinfo/nginx" rel="noreferrer" target="_blank">http://mailman.nginx.org/<wbr>mailman/listinfo/nginx</a><br></blockquote></div><br></div>