HTTP downloads randomly get stuck until client timeout

[Richard Stanway]

Hi all,
I'm experiencing odd behavior with some larger HTTP file downloads from my
site. The files will download for a seemingly random amount of bytes then
the connection freezes until "send_timeout" expires, at which point the
error log shows "client timed out (110: Connection timed out) while sending
response to client".

A tcpdump shows both ends successfully passing packets with no packet loss.
nginx is pretty stock from the nginx.org repository, no 3rd party modules
or complex options, mostly static files and fastcgi / PHP. The behavior is
very intermittent, but happens regardless of client browser / IP /
requested file.

I was able to captured a debug log when this happened, it is available at
https://hastebin.com/tevusuhobe.m (some rewrite details and variables have
been omitted)

# nginx -V
nginx version: nginx/1.10.2
built by gcc 4.9.2 (Debian 4.9.2-10)
built with OpenSSL 1.0.1t  3 May 2016
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx
--modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf
--error-log-path=/var/log/nginx/error.log
--http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid
--lock-path=/var/run/nginx.lock
--http-client-body-temp-path=/var/cache/nginx/client_temp
--http-proxy-temp-path=/var/cache/nginx/proxy_temp
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp
--http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx
--with-file-aio --with-threads --with-ipv6 --with-http_addition_module
--with-http_auth_request_module --with-http_dav_module
--with-http_flv_module --with-http_gunzip_module
--with-http_gzip_static_module --with-http_mp4_module
--with-http_random_index_module --with-http_realip_module
--with-http_secure_link_module --with-http_slice_module
--with-http_ssl_module --with-http_stub_status_module
--with-http_sub_module --with-http_v2_module --with-mail
--with-mail_ssl_module --with-stream --with-stream_ssl_module
--with-cc-opt='-g -O2 -fstack-protector-strong -Wformat
-Werror=format-security -Wp,-D_FORTIFY_SOURCE=2'
--with-ld-opt='-Wl,-z,relro -Wl,-z,now -Wl,--as-needed'

sendfile, tcp_nopush, tcp_nodelay are enabled.
accept_mutex, aio is disabled.

Linux karak 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19)
x86_64 GNU/Linux

I've also tried mainline nginx/1.11.9 from the nginx.org repository and the
problem persisted. Any advice on what I should be looking at to resolve
this would be very welcome!

Regards,

Richard.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20170131/7ea08004/attachment.html>