Response stalls with gzip and proxy

[technocoreai]

We've been experiencing a weird issue with nginx used as a gzipping proxy.
One of our internal services doesn't support content-encoding and because it
usually streams huge amounts of json data in response, we've decided to put
it behind nginx. Unfortunately, about 5-10% of responses in this
configuration seem to just stall with the TCP connection perfectly fine, but
no data going anywhere.

I've tried to diagnose the issue to check if it's a problem with our HTTP
libraries, VPN or network, but it does look like nginx is the culprit:
 - The issue is reproducible with any HTTP client, including curl
--compress.
 - Packet capture on the server that's hosting a client and the server
that's hosting nginx show nothing interesting. There are no lost or
corrupted packets; the client receives the last packet sent by nginx, ACKs
it and then ACKs keepalives. If the client is killed, it and nginx exchange
FIN/FIN-ACKs just fine.
 - Packet capture on the loopback interface, however, shows that nginx just
stops reading data at some point (I see a “full window” notice in wireshark
and netstat shows an nginx  worker sitting with a 512k-2M receive queue with
an empty transmit queue).

Some technical details (I've masked our internal host names):

The server on which nginx is on is running a server build of Ubuntu; uname
-a gives Linux <hostname> 3.2.0-32-generic #51-Ubuntu SMP Wed Sep 26
21:33:09 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux. This issue occurs on nginx
from the Ubuntu repo (1.1.19-1ubuntu0.1) and Debian wheezy repo
(1.2.1-2.2).

nginx -V output:
nginx version: nginx/1.2.1
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --conf-path=/etc/nginx/nginx.conf
--error-log-path=/var/log/nginx/error.log
--http-client-body-temp-path=/var/lib/nginx/body
--http-fastcgi-temp-path=/var/lib/nginx/fastcgi
--http-log-path=/var/log/nginx/access.log
--http-proxy-temp-path=/var/lib/nginx/proxy
--http-scgi-temp-path=/var/lib/nginx/scgi
--http-uwsgi-temp-path=/var/lib/nginx/uwsgi --lock-path=/var/lock/nginx.lock
--pid-path=/var/run/nginx.pid --with-pcre-jit --with-debug
--with-http_addition_module --with-http_dav_module --with-http_geoip_module
--with-http_gzip_static_module --with-http_image_filter_module
--with-http_realip_module --with-http_stub_status_module
--with-http_ssl_module --with-http_sub_module --with-http_xslt_module
--with-ipv6 --with-sha1=/usr/include/openssl --with-md5=/usr/include/openssl
--with-mail --with-mail_ssl_module
--add-module=/tmp/buildd/nginx-1.2.1/debian/modules/nginx-auth-pam
--add-module=/tmp/buildd/nginx-1.2.1/debian/modules/nginx-echo
--add-module=/tmp/buildd/nginx-1.2.1/debian/modules/nginx-upstream-fair
--add-module=/tmp/buildd/nginx-1.2.1/debian/modules/nginx-dav-ext-module

Basic nginx config is unchanged from Ubuntu-provided defaults; the server
section looks like this:

    server {
        gzip  on;
        gzip_proxied any;
        gzip_types application/json;
        
        listen 80 default;
        server_name <servername>;
    
        access_log  /var/log/nginx/<servername>-access.log;
    
        location / {
            proxy_pass http://127.0.0.1:8083;
            proxy_buffering off;
        }
    }

You can see the nginx debug log here: http://pastebin.com/qDis2RS4 (I've
masked the request URL but it's otherwise unchanged); it ends about 1 second
after the response stall.

Posted at Nginx Forum: http://forum.nginx.org/read.php?2,234275,234275#msg-234275