upstream timeouts I can not explain

[Руслан Закиров]

Hello,

nginx 1.10.2 [1] running on FreeBSD 10.3-RELEASE-p7

We have quite a lot of "upstream timed out" errors. For example:

2017/01/08 16:20:54 [error] 82154#0: *494223064 upstream timed out (60:
Operation timed out) while connecting to upstream, client: 192.168.1.43,
server: q0.sports.ru, request: "POST /pub?topic=docs HTTP/1.1", upstream: "
http://192.168.1.206:4151/pub?topic=docs", host: "q0.sports.ru"
Here is tcpdump:

16:20:53.218916 IP myself-fe.28957 > q0-2.4151: Flags [S], seq 2479999411,
win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 787079162 ecr 0],
length 0
16:20:53.219721 IP q0-2.4151 > myself-fe.28957: Flags [S.], seq 605231398,
ack 2479999412, win 28960, options [mss 1460,sackOK,TS val 352522447 ecr
787079162,nop,wscale 7], length 0
16:20:53.219742 IP myself-fe.28957 > q0-2.4151: Flags [.], ack 1, win 1040,
options [nop,nop,TS val 787079163 ecr 352522447], length 0
16:20:54.572712 IP myself-fe.28957 > q0-2.4151: Flags [F.], seq 1, ack 1,
win 1040, options [nop,nop,TS val 787080516 ecr 352522447], length 0
16:20:54.573173 IP q0-2.4151 > myself-fe.28957: Flags [F.], seq 1, ack 2,
win 227, options [nop,nop,TS val 352522786 ecr 787080516], length 0
16:20:54.573195 IP myself-fe.28957 > q0-2.4151: Flags [.], ack 2, win 1040,
options [nop,nop,TS val 787080516 ecr 352522786], length 0

What I don't see here is a packet with HTTP request. I checked access log
and only two requests to .206 upstream were finished at "16:20:54". Another
request looked like this:

16:20:54.026151 IP myself-fe.29450 > q0-2.4151: Flags [S], seq 347540799,
win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 787079969 ecr 0],
length 0
16:20:54.026502 IP q0-2.4151 > myself-fe.29450: Flags [S.], seq 2176380567,
ack 347540800, win 28960, options [mss 1460,sackOK,TS val 352522649 ecr
787079969,nop,wscale 7], length 0
16:20:54.026526 IP myself-fe.29450 > q0-2.4151: Flags [.], ack 1, win 1040,
options [nop,nop,TS val 787079970 ecr 352522649], length 0
16:20:54.026537 IP myself-fe.29450 > q0-2.4151: Flags [P.], seq 1:205, ack
1, win 1040, options [nop,nop,TS val 787079970 ecr 352522649], length 204
16:20:54.026779 IP q0-2.4151 > myself-fe.29450: Flags [.], ack 205, win
235, options [nop,nop,TS val 352522649 ecr 787079970], length 0
16:20:54.027136 IP q0-2.4151 > myself-fe.29450: Flags [P.], seq 1:119, ack
205, win 235, options [nop,nop,TS val 352522649 ecr 787079970], length 118
16:20:54.027145 IP q0-2.4151 > myself-fe.29450: Flags [F.], seq 119, ack
205, win 235, options [nop,nop,TS val 352522649 ecr 787079970], length 0
16:20:54.027156 IP myself-fe.29450 > q0-2.4151: Flags [.], ack 120, win
1038, options [nop,nop,TS val 787079970 ecr 352522649], length 0
16:20:54.027179 IP myself-fe.29450 > q0-2.4151: Flags [F.], seq 205, ack
120, win 1040, options [nop,nop,TS val 787079970 ecr 352522649], length 0
16:20:54.027354 IP q0-2.4151 > myself-fe.29450: Flags [.], ack 206, win
235, options [nop,nop,TS val 352522649 ecr 787079970], length 0

It has packet with HTTP request after establishing TCP connection.

Any idea where I should look next? Is it some known issue?

[1] nginx -V

nginx version: nginx/1.10.2
built with OpenSSL 1.0.2j  26 Sep 2016
TLS SNI support enabled
configure arguments: --prefix=/usr/local/etc/nginx --with-cc-opt='-I
/usr/local/include' --with-ld-opt='-L /usr/local/lib'
--conf-path=/usr/local/etc/nginx/nginx.conf
--sbin-path=/usr/local/sbin/nginx --pid-path=/var/run/nginx.pid
--error-log-path=/var/log/nginx/error.log --user=www --group=www
--with-file-aio
--http-client-body-temp-path=/var/tmp/nginx/client_body_temp
--http-fastcgi-temp-path=/var/tmp/nginx/fastcgi_temp
--http-proxy-temp-path=/var/tmp/nginx/proxy_temp
--http-scgi-temp-path=/var/tmp/nginx/scgi_temp
--http-uwsgi-temp-path=/var/tmp/nginx/uwsgi_temp
--http-log-path=/var/log/nginx/access.log --with-http_dav_module
--with-http_geoip_module --with-http_gzip_static_module
--with-http_realip_module --with-http_slice_module
--with-http_stub_status_module --with-http_sub_module --with-pcre
--with-http_v2_module --with-stream --with-stream_ssl_module
--with-http_ssl_module

-- 
Руслан Закиров
Руководитель отдела разработки веб-сервисов
+7(916) 597-92-69, ruz @  <http://www.sports.ru/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20170108/30962d3e/attachment.html>